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PREFACE 


This  paper  was  prepared  by  the  Institute  for  Defense  Analyses  (IDA)  for  the  Strategic 
Defense  Initiative  Organization  (SDIO)  as  a  follow-on  effort  for  Subtask  Order  T-R2- 
597.21,  “Software  Testing  of  Strategic  Defense  Systems.”  The  objective  of  this  subtask  is 
to  assist  the  SDIO  in  planning,  executing,  and  monitoring  software  testing  and  evaluation 
research,  development,  and  practice. 

In  support  of  this  objective,  IDA  conducted  an  examination  of  27  tools  that  support  soft¬ 
ware  testing.  These  tools  provide  for  test  management,  problem  reporting,  and  static  and  dy¬ 
namic  analysis  of  Ada  code.  This  paper  presents  the  results  of  the  examination  and  provides 
software  development  managers  with  information  on  current  capabilities  of  available  test¬ 
ing  tools. 

This  paper  was  reviewed  by  the  following  members  of  the  EDA  research  staff:  Dr.  Rob¬ 
ert  J.  Atwell,  Dr.  Dennis  W.  Fife,  Dr.  Randy  L.  Garrett,  Ms.  Deborah  Heystek,  Ms.  Audrey 
A.  Hook,  Dr.  Richard  J.  Ivanetich,  Dr.  Reginald  N.  Meeson,  and  Dr.  Richard  L.  Wexelblat. 
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SUMMARY 


Software  testing  is  labor  intensive  and  can  consume  over  50%  of  software  development 
costs.  Rarely  is  sufficient,  effective  testing  performed  as  evidenced  by  the  fact  that  a  failure 
rate  of  3  to  10  failures  per  thousand  lines  of  code  is  typical  for  commercial  software.  More¬ 
over,  the  cost  of  correcting  a  defect  increases  as  software  development  progresses;  for  ex¬ 
ample,  the  cost  of  fixing  a  requirements  fault  during  operation  can  be  60  to  100  times  the 
cost  of  fixing  that  same  fault  during  early  development  stages.  Consequently,  timely  defect 
detection  is  important.  Automated  testing  tools  can  alleviate  these  problems  by  providing 
managers  with  more  insight  into  the  progress  of  test  activities,  by  reducing  the  traditionally 
manual  nature  of  testing,  and  by  encouraging  the  application  of  improved  testing  practices. 
Yet  reviews  of  testing  practices  and  tool  usage  reveal  extremely  poor  exploitation  of  avail¬ 
able  testing  tool  support  Recent  surveys  of  test  practitioners  indicate  that  there  are  few 
common  test  practices  and  only  scattered  tool  usage. 

Over  600  testing  tools  from  some  400  suppliers  were  identified  during  the  course  of  this 
study.  From  these,  27  tools  were  selected  for  examination.  These  tools  support  test  manage¬ 
ment,  problem  reporting,  and  static  and  dynamic  analysis  of  Ada  code.  Consideration  of 
tools  that  are  dependent  on  special  hardware,  limited  to  regression  analysis,  or  form  an  in¬ 
tegral  part  of  a  computer-aided  software  engineering  (CASE)  system  was  postponed  for  a 
later  effort.  Also,  care  was  taken  not  to  duplicate  the  tool  assessment  efforts  of  other  groups. 
During  the  course  of  the  examination,  the  static  and  dynamic  analysis  tools  were  applied  to 
a  series  of  Ada  programs  in  order  to  assess  their  functionality.  The  test  management  and 
problem  reporting  tools  were  also  subject  to  a  practical  examination  using  vendor-supplied 
data.  Each  tool  was  then  described  in  terms  of  its  functionality,  ease  of  use,  and  documen¬ 
tation  and  support.  Problems  encountered  during  tool  use  and  other  pertinent  observations 
were  also  recorded. 

Significant  findings  from  this  study  include  the  following: 

•  Test  management  tools  offer  critically  needed  support  for  test  planning  and  test 
progress  monitoring.  This  category  of  test  tool  is  perhaps  the  latest  to  come  to  mar¬ 
ket  With  the  exception  of  reliability  analysis  tools,  which  are  becoming  more  com¬ 
mon,  progress  monitoring  capabilities  are  infrequently  available  and  primitive. 
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Nevertheless,  the  ability  of  these  tools  to  manage  a  collection  of  test  information  is 
very  valuable  and  the  data  available  from  the  analysis  of  this  information  is  urgently 
needed  to  support  the  documentation  and  management  of  test  activities. 

•  Problem  reporting  tools  offer  additional  support  for  test  management  by  provid¬ 
ing  insight  into  the  status  of  software  products  and  the  progress  of  development 
activities.  These  tools  are  primarily  intended  to  support  the  tracking  of  identified  soft¬ 
ware  problems  and  the  management  of  problem  resolution.  They  also  provide  infor¬ 
mation  on  the  status  and  quality  of  software  products;  in  particular,  they  capture  the 
data  needed  for  software  reliability  modeling.  This  data  can  also  provide  valuable 
insights  into  the  status  and  quality  of  the  software  development  processes  themselves, 
and  so  support  continuous  process  improvement 

•  Available  static  analysis  tools  are  essentially  limited  to  facilitating  program 
understanding  and  assessing  characteristics  of  software  quality.  They  provide 
some  minimal  support  for  guiding  dynamic  testing.  The  types  of  defects  tradition¬ 
ally  found  by  static  analysis  tools  are  now  routinely  checked  for  by  Ada  compilers. 
Currently,  complexity  analysis,  control  flow  analysis,  and  software  browsing  are  the 
most  common  static  analysis  functions  supported. 

•  Although  many  needed  dynamic  analysis  capabilities  are  not  commonly  avail¬ 
able,  tools  are  available  that  offer  considerable  support  for  dynamic  testing  to 
increase  confidence  in  correct  software  operation.  Dynamic  analysis  is  the  princi¬ 
ple  method  used  for  software  validation  and  verification.  Here  automated  support  for 
the  preparation  of  a  test  bed,  generation  of  test  data,  and  analysis  of  test  results  is 
urgently  needed.  Tools  that  provide  this  functionality  will  decrease  the  cost  of  testing 
by  increasing  the  productivity  of  the  human  tester  and  increase  software  quality  by 
supporting  test  data  adequacy  analysis  and  test  repeatability.  Tools  that  offer  elements 
of  this  type  of  support  are  available. 

Tools  of  similar  types  vary  widely  in  the  capabilities  they  provide  and  in  characteristics 
such  as  tailorability  and  robustness.  Existing  testing  tools  are  generally  easy  to  use  and  sup¬ 
ported  by  good  documentation.  There  were  instances  during  this  study,  however,  where  dif¬ 
ferent  tools  gave  different  results  when  performing  the  same  function,  for  example, 
calculating  cyclomatic  complexity.  Moreover,  some  of  the  tools  contained  faults.  While 
most  failures  were  trivial,  others  rendered  a  tool  unusable. 

Available  testing  tools  offer  important  opportunities  for  increasing  software  quality  and 
reducing  development  and  support  costs.  Even  so,  there  are  a  number  of  specific  problems 
with  these  tools  and  a  lack  of  needed  functionality  that  may  handicap  testing  of  Ada  soft¬ 
ware: 

•  There  is  a  lack  of  tool  support  for  testing  concurrent  Ada  software. 

•  There  is  a  need  for  increased  tool  integration  to  provide  more  complete  coverage  of 
testing  activities. 


There  is  a  need  for  integration  of  testing  tools  into  CASE  systems  to  provide 
improved  feedback  into  development  activities. 

There  is  a  lack  of  data  on  the  cost  effectiveness  of  particular  test  techniques  and  tools 
that  can  be  used  to  encourage  and  guide  tool  use. 
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STUDY  OVERVIEW 


PARTI 


Introduction 


1.  INTRODUCTION 


1.1  Purpose 

This  report  provides  software  developers  with  information  that  will  help  them  gain  an 
understanding  of  the  types  of  software  testing  tools  that  are  available,  the  functionality  of 
these  tools,  and  how  they  can  aid  the  development  and  support  of  Ada  software  for  the  Stra¬ 
tegic  Defense  Initiative  (SDI). 


1.2  Scope 

Tools  are  available  to  support  a  variety  of  testing  tasks  at  different  stages  in  the  software 
life  cycle.  To  make  best  use  of  available  resources,  the  work  described  here  was  initially 
limited  to  the  examination  of  tools  that  support  the  static  and  dynamic  analysis  needed  for 
testing  Ada  code.  Code-based  testing  was  selected  as  being  one  area  where  automated  sup¬ 
port  is  critically  needed,  both  to  increase  software  reliability  and  to  reduce  development  and 
support  costs.  Restriction  to  the  Ada  programming  language  [ANSI/MIL-STD-1983]  was 
adopted  in  view  of  Department  of  Defense  (DoD)  Instruction  5000.2  [DoDI  1991].  The 
scope  of  the  study  was  subsequently  extended  to  include  test  management  and  problem  re¬ 
porting  tools.  The  purpose  of  this  extension  is  to  accommodate  DoD’s  increasing  trend  to¬ 
wards  the  use  of  software  metrics  to  support  the  management  of  software  development  and 
as  a  basis  for  continual  process  improvement 

The  report  is  divided  into  two  parts.  Part  I  starts  by  setting  the  scene  for  the  following 
discussions  by  taking  a  brief  look  at  the  current  state  of  practice  in  software  testing.  Special 
software  test  requirements  imposed  by  the  Strategic  Defense  Initiative  Organization 
(SDIO),  and  how  automated  test  tools  could  support  meeting  these  requirements,  are  also 
discussed.  Part  I  goes  on  to  describe  how  particular  tools  were  selected  for  examination, 
identifies  the  tools  so  selected,  and  outlines  the  method  of  examination.  The  following  sec¬ 
tions  summarize  tool  functionality  in  the  areas  of  test  management,  problem  reporting,  stat¬ 
ic  analysis,  and  dynamic  analysis.  This  first  part  of  the  report  concludes  by  summarizing  the 
findings  resulting  from  this  work. 

Based  on  the  experience  gained  during  their  examination.  Part  II  provides  a  usage-based 
description  of  the  tools  and  example  report  outputs.  This  more  technical  presentation  is  in¬ 
tended  to  provide  further  insight  for  the  potential  tool  user. 
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This  is  a  follow-on  report  to  IDA  Paper-2686  [Youngblut  1991].  The  earlier  report  dis¬ 
cusses  the  examination  of  some  10  tools  for  the  static  and  dynamic  analysis  of  Ada  code. 
For  convenience,  those  discussions  have  been  updated  as  appropriate  and  are  included  here. 


PARTI 


State  of  Practice 


2.  STATE  OF  PRACTICE 


The  high  cost  of  software  testing  has  long  been  recognized  by  the  software  community. 
In  the  early  1970s,  data  collected  during  development  of  a  number  of  large  software  systems 
(e.g.,  SAGE,  NTDS,  Gemini,  Saturn  V,  and  IBM  OS/360)  revealed  that  50%  of  develop¬ 
ment  costs  were  incurred  by  software  testing  [Boehm  1980].1  This  figure  holds  true  today 
[AFSCP  1987,  Korel  1991,  Yourdon  1990].  Even  with  this  level  of  effort,  operational  soft¬ 
ware  still  fails.  Commercial  software  typically  experiences  3  to  10  failures  per  thousand  line 
of  code  (KLOC)  and  industrial  software  experiences  1  to  3  failures  per  KLOC  [Boehm 
1988]. 

Recent  surveys  of  current  testing  practices  help  to  explain  these  figures.  The  Software 
Test  Practices  Survey  [SQE  1990],  conducted  at  the  Seventh  International  Software  Testing 
Conference,  for  example,  found  that  software  test  practices  were  weak  at  the  unit  testing 
level  and  only  slightly  better  for  system  and  acceptance  testing.  In  fact,  when  common  test¬ 
ing  practices  were  defined  as  those  which  more  than  60%  of  respondents  ranked  as  standard 
practice,  no  common  practices  for  unit  testing  could  be  identified.  Table  2-1  shows  the  per¬ 
centage  of  responses  indicating  testing  process  and  management  practices  as  standard. 

Table  2-1.  Practices  Reported  In  Software  Test  Practices  Survey 


Percentage  of  responses  Indicating  practices 
common  or  standard 

fest 

System  Test 

Process  Practices 

■ 

Software  risks  are  systematically  analyzed. 

Test  cases  &  procedures  are  formally  documented. 

Test  are  specified  before  software  design. 

11 

25 

6 

B 

n 

Test  cases  &  procedures  are  saved  after  testing. 

29 

58 

54 

Formal  report  of  test  results  is  produced. 

33 

<5 

60 

Requirements  coverage  is  analyzed  or  traced. 

17 

55 

54 

Code  coverage  is  analyzed  or  traced. 

18 

28 

27 

Design  coverage  is  analyzed  or  traced. 

15 

32 

38 

Formal  exit  criteria  used  to  specify  test  completion. 

22 

18 

17 

Tests  are  rerun  after  software  changes. 

Test  process  is  systematic  and  standardized. 

18 

39 

38 

39 

70 

65 

Test  cases  &  procedures  assigned  unique  names. 

Management  Practices 

21 

55 

54 

Boldface  print 

A  record  of  time  spent  on  testing  is  produced. 

14 

42 

39 

indicates  common 

Cost  of  testing  is  measured  and  reported. 

11 

28 

24 

practice  (>60%) 

A  record  of  faults  and  defects  found  is  produced. 

26 

68 

70 

The  patterns  of  faults  and  defects  regularly  analyzed. 

10 

27 

24 

Defect  density  is  measured. 

10 

19 

16 

User  or  customer  satisfaction  is  measured. 

17 

42 

42 

Number  of  changes  or  change  requests  is  measured. 

8 

16 

16 

Test  effectiveness  and  efficiency  measured  &  reported. 

KJ 

20 

21 

Testers  are  formally  trained. 

m 

30 

23 

The  test  process  is  documented  in  standards  manual. 

E2 

22 

30 

1.  For  NASA’s  Apollo  program,  80%  of  the  total  software  development  effon  was  incurred  by  test¬ 
ing  [Dunn  1984]. 
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Xerox  Corporation  and  Software  Quality  Engineering  conducted  a  joint  survey  called 
the  Software  Measures  and  Practices  Benchmark  Study  [SQE  1991].  The  first  part  of  this 
work  provided  a  preliminary  assessment  of  typical  software  practices  and  measures  in  use 
in  industry.  The  results  of  this  initial  work  were  used  to  identify  those  organizations  that 
employ  the  most  of  what  industry  generally  considers  to  be  good  practices.  The  organiza¬ 
tions  selected  were  AT&T,  E.I.  DuPont  de  Nemours,  GTE  Corporation,  IBM,  NCR  Corpo¬ 
ration,  Siemens  AG,  and  Xerox  Corporation.  Each  was  asked  to  pick  one  or  two  of  their 
“best”  projects  from  which  to  provide  data  for  the  survey.  Table  2-2  reproduces  some  of 
the  resuits.  Even  though  these  organizations  were  selected  as  ones  that  most  frequently  em¬ 
ploy  advanced  testing  practices,  very  few  testing  practices  were  in  common  use  at  that  time. 

Although  tools  are  more  frequently  used  for  system  and  acceptance  testing  rather  than 
unit  testing,  the  Software  Test  Practices  Survey  found  that  there  were  no  types  of  tools  that 
more  than  60%  of  respondents  cited  as  commonly  used.  As  shown  in  Figure  2-1  the  most 
widely  used  type  of  tool  was  only  used  by  50%  of  the  respondents.  Similarly,  the  Software 
Measures  and  Practices  Benchmark  Study  found  only  scattered  use  of  tools. 


Table  2-2.  Practices  Reported  In  Software  Measures  &  Practices  Benchmark  Survey 


Mean  scons  for  practical  usage 

Process  Practices 

1 

Low. 

1 1  P-T.TT! 

High  1 

Software  risks  (potential  failures)  are  systematically  analyzed. 

1.00 

137 

1.94 

Test  planning  &  specifications  are  stated  in  requirements  phase. 

.90 

1.47 

2.14 

Unit  test  plans/specificabons  are  prepared. 

1.19 

1.96 

233 

Someone  other  than  programmer  perfotms/reviews  unit  testing. 

.87 

134 

239 

Module  or  program  complexity  is  measured. 

.46 

1.16 

1.81 

Software  changes  are  analyzed  for  ripple  effect  and  test  impact 

1.47 

1.97 

233 

Unit  branch  &  statement  execution  coverage  is  analyzed. 

134 

1.49 

Unit  test  results  are  recorded. 

1.89 

2.71 

Tests  are  cross-referenced  to  requirements. 

Test  plans  &  specifications  are  formally  reviewed. 

1.65 

2.13 

1.91 

231 

236 

Code  coverage  is  analyzed  for  entire  system  during  system  test 

.68 

131 

2.14 

Random  testing  is  used  to  evaluate  reliability. 

1.00 

1.72 

2.26 

Tests  are  systematically  saved  &  reused. 

1.10 

2.18 

2.83 

Boldface  print 

Features  fixed  in  previous  test  cycles  systematically  retested. 

2.21 

2.67 

indicates  common 

Management  Practices 

practice  (>235) 

Cost  of  quality  activities  is  measured  and  reported. 

37 

137 

2.41 

Defects  are  analyzed  to  determine  cause  &  when  created. 

nrrn 

135 

2.02 

Defects  found  during  testing  are  recorded  &  tracked. 

2.42 

2.70 

239 

Defect  analysis  &  trends  used  to  identify  process  changes. 

IKOM 

1.47 

1.96 

Number  of  defects  found  after  release  is  measured. 

1.62 

2.62 

3.00 

Number  of  new  defects  introduced  per  “fix”  is  recorded. 

133 

1.87 

2.91 

Time  to  identify  &  correct  defects  is  measured. 

2.91 

Test  procedures  &  policies  are  clearly  identified  &  described. 

1.86 

Score  Usage 
<  1 .25  Uncharacteristic 


1.25-1.75  Scattered 
1.75-2.25  Significant 
>2.25  Accepted 
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Figure  2-1.  Tool  Usage  Reported  In  Software  Test  Practices  Survey 


Although  the  results  from  only  two  surveys  are  cited  here,  there  is  much  data  to  support 
these  findings.  Similar  data  is  provided  in,  for  example,  the  survey  sponsored  by  the  Mas¬ 
sachusetts  Computer  Software  Council  [KPMG  1992]. 
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3.  TEST  REQUIREMENTS  AFFECTING  TOOL  USE 

This  section  considers  three  major  drivers  that  encourage  the  use  of  testing  tools  for  the 
development  of  SDI  software.  The  first  of  these  is  the  current  set  of  SDIO  documents  that 
provides  policy  and  guidance  for  software  development  in  general.  The  second  driver  is  the 
Software  Engineering  Institute  (SEI)  Process  Maturity  Model  (PMM)  [Humphrey  1987] 
that  routinely  will  be  used  in  the  near  future  to  conduct  evaluations  of  SDI  contractors’  soft¬ 
ware  engineering  practices.  The  final  driver  considered  is  the  Global  Protection  Against 
Limited  Strikes  (GPALS)  Computer  Resources  Working  Group  (CRWG)  software  metrics 
evaluation  program. 

SDIO  requirements  are  not  the  only  reason  to  use  automated  test  tools.  Indeed,  because 
of  the  complexity  of  detail  involved  in  testing  even  the  simplest  program,  tools  are  a  pre¬ 
requisite  for  most  forms  of  static  and  dynamic  analysis.  Similarly,  the  ability  to  capture,  an¬ 
alyze,  and  present  quantitative  process  measurement  data  in  a  meaningful  form  greatly 
facilities  test  management.  Although  there  is  a  lack  of  consilient  data  on  the  cost  effective¬ 
ness  of  particular  testing  tools,  there  can  be  no  doubt  that  automated  tools  are  able  to  im¬ 
prove  the  cost  effectiveness  of  testing.  On?  test  practitioner,  for  example,  cites  reductions 
in  testing  time  of  up  to  70%,  a  30%  increase  in  overall  software  development  productivity 
[Graham  1991]. 

3.1  Affect  of  SDIO  Software  Test  Requirements  on  Tool  Use 

SDIO  encourages  the  use  of  automated  tools  to  support  testing.  Candidate  tool  classes 
identified  in  the  Global  Protection  Against  Limited  Strikes  (GPALS)  Software  Standards 
[GPALS  1992c],  for  example,  are  test  case  generators,  performance  analyzers,  complexity 
analyzers,  and  regression  analysis  tools.  The  use  of  source  code  standards  checking,  formal 
verification,  and  static  and  dynamic  code  analysis  tools  is  also  discussed.  The  Trusted  Soft¬ 
ware  Guide  annex  to  the  GPALS  Software  Standards  requires  the  use  of  an  automated  test¬ 
bed  for  creating,  executing,  documenting,  managing,  and  analyzing  the  completeness  of  all 
tests,  and  for  maintaining  test  documentation.  The  GPALS  Software  Quality  Program  Plan 
[GPALS  1992a]  requires  the  use  of  automated  metrics  data  collection  and  reporting  tools. 

Additionally,  the  SDIO  Software  Policy  [SDIO  1992b]  and  the  SDIO  Contract 
Requirements  Packages  (CRPs)  Guidelines  for  Computer  Resource  Issues  [GPALS  1992b] 
impose  requirements  on  testing  practices  that,  either  directly  or  indirectly,  foster  tool  use. 
These  special  requirements  and  their  sources  are  identified  in  Table  3-1.  litis  table  also 
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Table  3-1.  SDIO  Test  Requirements 


TEST  REQUIREMENT 

SDIO  Software  Policy 

GPALS  Sw.  Standards 

GPALS  Sw.  Trust  Guide 

GPALS  SQPP 

GPALS  CRPs 

POSSIBLE  TOOL 
SUPPORT 

Continuous  process  improvement.  Use  of  concurrent  engineer¬ 
ing  practices  to  provide  continuous  improvement  in  software  engi¬ 
neering  processes  and  the  visible  quality  of  products. 

1 

1 

1 

1 

1 

Problem  reporting,  reliability 
analysis,  cost  analysis,  progress 
monitoring 

Quality  evaluation.  Data  collection  and  reporting  of  a  minimum 
set  of  software  process,  product,  and  management  metrics. 

1 

H 

H 

Quality  analysis 

Minimum  structural  test  coverage. 

i)  Structural  test  coverage  for  CSU/CSCI  and  regression  testing  of 
all  statement,  branches,  loops. 

ii)  Structural  coverage  and  boundary  value  testing  at  the  unit  level, 
demonstration  of  coverage  at  integration  level. 

B 

1 

< 

Structural  coverage  analysis 

Structural  coverage  analysis 

Test  traceability.  Traceability  of  requirements,  design,  and  code  to 
tests  and  test  results. 

1 

H 

H 

1 

H 

Requirements  tracking,  test 
planning 

Design  and  code  inspections.  Formal  inspections  for  all  software 
designs  and  code  products. 

1 

H 

1 

1 

H 

Browsing 

Review.  Review  of  CSU  tests  and  results. 

■ 

H 

D 

■ 

Progress  monitoring 

Testable  requirements.  Demonstration  of  an  objective  and  feasi¬ 
ble  test  of  whether  each  requirement  is  met. 

1 

1 

1 

1 

H 

Requirements  tracking,  test 
planning 

Functional  testing.  The  process  of  exercising  a  system  under 
operational  conditions  to  determine  that  specified  functional 
requirements  are  implemented  correctly. 

1 

1 

1 

i 

i 

Test  data  &  testbed  generation, 
functional  coverage  analysis 

Reliability  measurement.  Statistical  techniques  used  to  reduce 
observed  software  defects  to  acceptable  limits. 

1 

1 

H 

1 

1 

Problem  reporting,  reliability 
analysis 

Random  testing.  In  addition  to  other  methods  for  generating  test 
input,  random  input  generated  to  overcome  any  test  bias. 

1 

1 

H 

1 

1 

Test  data  &  testbed  generation 

Penetration  testing.  Penetration  tests  required  as  part  of  establish¬ 
ing  software  trust. 

1 

1 

H 

1 

1 

~ 

Regression  testing.  Retest  modified  software  to  verify  that 
changes  have  not  caused  unintended  effects  and  software  still 
meets  the  requirements. 

1 

1 

i 

i 

i 

Regression  &  change  analysis, 
requirements  tracking,  test 
planning 

Test  progress  tracking.  Progress  tracked  and  compared  to  the 
Software  Test  Plan. 

1 

1 

1 

1 

1 

Test  planning,  cost  analysis, 
progress  monitoring,  problem 
reporting,  requirements  track¬ 
ing 

Static  and  dynamic  code  analysis.  Complexity,  structure,  and 
style  assessment,  and  checking  for  language  violations,  unused 
code  or  data. 

1 

1 

i 

i 

i 

Static  and  dynamic  code  analy¬ 
sis,  test  data  &  testbed  genera¬ 
tion 

Source  code  standards  compliance.  Code  portability  and  style 
assessment 

1 

H 

■ 

■ 

■ 

Auditing,  complexity  analysis, 
structure  analysis 

Test  repeatability.  The  ability  to  repeat  a  test  with  the  same  inputs 
and  operating  conditions  to  yield  the  same  results. 

1 

H 

H 

■ 

■ 

Test  planning  &  documenta¬ 
tion,  testbed  generation 
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identifies  the  types  of  testing  tools  that  can  be  expected  to  increase  significantly  the  cost 
effectiveness  of  the  associated  test  activities. 


3.2  Affect  of  the  SEI  Process  Maturity  Model  on  Tool  Use 

Starting  in  FY93,  SDIO  will  require  evaluation  of  contractor  software  engineering  ca¬ 
pabilities  using  the  SEI  PMM.  This  evaluation  will  be  routinely  conducted  as  part  of  source 
selection  activities,  and  yearly  during  the  course  of  a  contract,  by  an  independent  team  of 
evaluators.  Contractors  will  be  encouraged  to  perform  annual  self-appraisals.  The  PMM  is 
used  to  rank  software  engineering  capabilities  as  the  following: 

•  Level  1  -  Initial.  The  software  process  is  characterized  as  ad  hoc,  and  occasionally 
even  chaotic.  Few  processes  are  defined  and  success  depends  on  individual  effort. 

•  Level  2  -  Repeatable.  Basic  project  management  processes  are  established  to  track 
cost,  schedule,  and  functionality.  The  necessary  process  discipline  is  in  place  to  repeat 
earlier  successes  on  projects  with  similar  applications. 

•  Level  3  -  Defined.  The  software  process  for  both  management  and  engineering  activ¬ 
ities  is  documented,  standardized,  and  integrated  into  an  organization-wide  software 
process.  All  projects  use  a  documented  and  approved  version  of  the  organization’s 
process  for  developing  and  maintaining  software. 

•  Level  4  -  Managed.  Detailed  measures  of  the  software  process  and  product  quality  are 
collected.  Both  the  software  process  and  products  are  quantitatively  understood  and 
controlled  using  detailed  measures. 

•  Level  5  -  Optimized.  Continuous  process  improvement  is  enabled  by  quantitative 
feedback  from  the  process  and  from  testing  innovative  ideas  and  technologies. 

As  part  of  the  evaluation,  the  PMM  queries  the  use  of  automated  requirements  trackers,  test 
data  generators,  coverage  analyzers,  complexity  analyzers,  cross-referencers,  and  interac¬ 
tive  source-level  debuggers.  The  testing-related  questions  that  are  asked  in  determining  the 
software  engineering  capability  level  are  listed  in  Table  3-2.  This  table  identifies  the  types 
of  testing  tools  that  could  be  used  to  support  the  identified  activities.  The  PMM  also  ad¬ 
dresses  the  use  of  process  and  product  measures  for  monitoring  the  status  and  quality  of 
both  the  developing  product  and  the  development  process.  In  this  case,  data  collected  in  the 
course  of  testing  activities  can  serve  several  purposes.  In  addition  to  supporting  the  deter¬ 
mination  of  the  effectiveness  of  actual  testing  activities,  this  data  provides  valuable  insight 
into  other  development  activities  such  as  defect  prevention,  training,  and  software  quality 
assurance.  Test-related  data  can  also  be  used  in  the  assessment  of  the  benefits  and  effec- 
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tiveness  of  existing  and  new  technology.  Test  tools  can  support  the  collection  of  much  of 
this  data. 

The  PMM  is  currently  being  revised.  The  new  version,  called  the  Capability  Maturity 
Model  [Paulk  1991],  extends  the  information  sought  on  testing  practices,  such  as  the  fol¬ 
lowing: 


Table  3-2.  PMM-Implied  Test  Requirements 


KEY  AREA 

PMM  QUESTIONS 

Process  Metrics 

Are  statistics  on  software  code  and  errors  gathered? 

Are  code  and  test  errors  projected  and  compared  to  actuals? 

Are  profiles  maintained  of  actual  versus  planned  software 
units  completing  unit  testing  over  time? 

Are  profiles  maintained  of  actual  versus  planned  software 
units  integrated  over  time? 

Is  test  coverage  measured  and  recorded  for  each  phase  of 
functional  testing? 

Are  software  trouble  reports  resulting  from  testing  tracked  to 
closure? 

Is  test  progress  tracked  by  deliverable  software  component 
and  compared  to  the  plan? 

Data  Management 
&  Analysis 

Is  error  data  from  code  reviews  and  tests  analyzed  to  deter¬ 
mine  the  likely  distribution  and  characteristics  of  the  errors 
remaining  in  the  product? 

Are  analyses  of  errors  conducted  to  determine  their  process- 
related  causes? 

Is  a  mechanism  used  for  error  cause  analysis? 

Is  software  productivity  analyzed  for  major  process  steps?- 

Process  control 

Is  there  a  mechanism  for  assuring  that  regression  testing  is 
routinely  performed? 

Is  there  a  mechanism  for  ensuring  the  adequacy  of  regression 
testing? 

Are  formal  test  case  reviews  conducted? 

Documented 
Standards  & 

Are  standards  applied  to  the  preparation  of  unit  test  cases? 

Are  coding  standards  applied  to  each  project? 

Procedures 

Are  formal  procedures  applied  to  estimating  software  develop¬ 
ment  schedules/cost? 

«  POSSIBLE 
,2  TOOL  SUPPORT 


2  Problem  reporting, 
static  analysis 


Problem  reporting, 
test  planning 


2  Progress  monitoring, 
test  planning 


2  Progress  monitoring, 
test  planning 


Coverage  analysis 


2  I  Problem  reporting 


2  Progress  monitoring, 
test  planning,  cost 
analysis 


Problem  reporting 


Problem  reporting 


Progress  monitoring 


3  Change  analysis, 
coverage  analysis 


3  DoD  document  gen¬ 
eration 


Auditing 


2  Progress  monitoring, 
test  planning,  cost 
analysis 
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•  Verification  of  software  requirements,  design,  and  code  according  to  the  project’s 
defined  software  process. 

•  Use  of  formal  criteria  to  determine  readiness  for  any  level  of  testing. 

•  Review  of  test  plan,  test  procedures,  and  test  cases  by  peers  of  the  developers  of  the 
plan  and  procedures  before  they  are  considered  ready  for  use. 

•  Appropriate  change  of  the  test  plan,  test  procedures,  and  test  cases  whenever  the  allo¬ 
cated  requirements,  software  requirements,  software  design,  or  code  being  tested 
changes. 

•  Determination  of  the  adequacy  of  testing  based  on  the  level  of  testing  performed,  the 
test  strategy  selected,  and  the  test  coverage  to  be  achieved. 

•  Performance  of  formal  system  testing  of  the  software,  according  to  the  project’s 
defined  software  process,  to  ensure  that  the  software  satisfies  the  software  require¬ 
ments. 

•  Performance  of  acceptance  testing  of  the  software,  according  to  the  project’s  defined 
software  process  and  approved  acceptance  test  plan,  to  demonstrate  to  the  customer 
and  end  users  that  the  software  satisfies  the  allocated  requirements. 

•  Maintenance  of  consistency  across  the  software  engineering  products,  including  the 
software  plans,  allocated  requirements,  software  requirements  specification,  software 
design,  code,  test  plans,  and  test  procedures. 

Here  again,  test  tools  can  be  expected  to  play  an  important  supporting  role. 


3.3  Affect  of  the  Software  Metrics  Program  on  Tool  Use 

No  set  of  metrics  for  software  project  management  has  gained  widespread  acceptance 
by  software  developers.  Accordingly,  the  GPALS  CRWG  on  Software  Quality  Improve¬ 
ment  and  Standards  (SQI&S)  has  developed  a  Software  Metrics  Evaluation  Plan  (SMEP) 
[SDIO  1992a]  designed  to  evaluate  and  provide  SDIO  with  recommendations  on  metrics 
and  metrics  tools  that  can  be  implemented  SDI-wide.  This  on-going  program  will  involve 
the  evaluation  of  several  sets  of  metrics  and  metrics  tools  on  a  number  of  different  SDI  soft¬ 
ware  development  projects.  The  first  evaluation  is  expected  to  proceed  through  1993  and 
will  be  conducted  on  the  SDI  Level  2  System  Simulator  (L2SS). 

The  SMEP  considers  three  functional  classes  of  metrics:  management,  process,  and 
product  metrics.  The  metrics  chosen  for  initial  evaluation  include  those  identified  by  the 
Army’s  Software  Test  and  Evaluation  Panel  (STEP)  [U.S.  Army  1992].  Metrics  from  the 
Air  Force’s  Software  Management  Indicators  [AFSCP  1986]  and  from  Martin  Marietta’s 
Pro-90  Software  Metrics  Handbook  [Martin  Marietta  1991]  will  be  used  to  estimate  com- 
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puter  resource  use.  Table  3-3  identifies  specific  SMEP  metrics  and  the  types  of  tools  that 
support  their  evaluation. 


Table  3-3.  Software  Metrics  Plan  Implied  Test  Requirements 


METRIC 

TYPE 

METRIC 

POSSIBLE  TOOL  SUPPORT 

CRWG 

Evaluated  Tool 
Support 

SASET 

ee 

u 

a. 

tw 

Analyze 

ADAMAT 

SQMS 

Sizing 

Cost  modeling 

El 

Management 

Costing,  Schedule,  Manloading 

Cost  modeling 

El 

Computer  Resource  Utilization 

- 

Requirements  Analysis 

Requirements  tracking 

Nonconformance  Reporting 

Problem  reporting 

El 

SDP  &  Software  Standards 

Auditing 

Utilization  of  Software  Tools 

Tool  inventorying 

Process 

Configuration  Management 

Change  control 

Change  Summary  Process 

Problem  reporting 

Productivity  Measures 

Progress  monitoring 

El 

El 

Development  Progress 

Progress  monitoring 

Cost 

Cost  analysis 

Defect  Density 

Problem  reporting 

El 

Maintainability 

Quality  analysis 

U 

El 

Cyclomatic  Complexity 

Complexity  analysis 

El 

El 

El 

I/O  Statements 

Quality  analysis,  static  analysis 

El 

n 

Product 

Entry  &  Exit  Points 

Quality  analysis,  static  analysis 

El 

Volume 

Complexity  analysis 

El 

El 

Portability 

Quality  analysis 

El 

Reliability 

Reliability  analysis 

El 

El 

Documentation 

Document  generation 

_ 

□ 

To  date,  the  CRWG  has  sponsored  the  examination  of  the  following  five  tools  to  assess 
their  support  for  the  application  of  the  SMEP  metrics  in  the  L2SS  evaluation  [Martin  Mari¬ 
etta  1992]: 

•  Software  Architecture,  Sizing,  and  Estimating  Tool  (SASET).  A  cost,  schedule,  and 
sizing  model  that  provides  software  development  estimates. 

•  Software  Problem  and  Change  Report  (SPCR).  Tracks  and  reports  on  nonconforming 
conditions  and  the  status  of  closure  and  corrective  actions. 
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Analyze.  Estimates  productivity  in  terms  of  the  ratio  of  the  number  of  executable 
lines  of  code  to  the  total  lines  of  source  code  and  collects  statistics  on  source  code. 
Ada  Measurement  and  Analysis  Tool  ( ADAMAT).  Collects  some  1 50  parameters  to 
estimate  software  reliability,  maintainability,  and  portability. 

Software  Quality  Management  System  (SQMS).  Collects  parameters  to  estimate  soft¬ 
ware  reliability,  complexity,  and  a  quality  index. 
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4.  APPROACH  AND  METHODS 

The  overall  approach  taken  to  this  work  was  to  identify  suppliers  of  testing  tools,  select 
tools  for  examination,  and  apply  the  selected  tools  in  the  testing  of  sample  pieces  of  code. 
The  tools  examined  to  date  are  all  available  independently  of  any  particular  computer-aided 
software  engineering  (CASE)  system  or  Ada  development  environment.  This  section  also 
summarizes  the  types  of  testing  support  provided  by  these  larger-scale  products  so  that  their 
testing  capabilities  can  be  contrasted  with  those  provided  by  the  independent  tools. 


4.1  Tool  Selection 

Nearly  four  hundred  suppliers  of  over  six  hundred  tools  were  identified.  From  this  ini¬ 
tial  set  of  suppliers,  a  short  list  was  prepared  of  those  tools  that  support  static  and  dynamic 
analysis  of  Ada  code,  test  management,  and  problem  reporting.  Information  was  sought 
from  the  appropriate  suppliers.  In  several  cases,  suppliers  gave  in-house  demonstrations  of 
their  tools.  Additional  criteria  were  then  applied  to  refine  the  short  list  to  be  compatible 
with  the  resources  available  for  tool  examination.  To  ensure  that  the  results  apply  to  the 
largest  possible  audience,  it  was  decided  that  selected  tools  should  be  essentially  indepen¬ 
dent  of  processor  architecture.  Consequently,  tools  such  as  non-intrusive  coverage  moni¬ 
tors  which  require  special  purpose  hardware  were  not  considered. 

Tool  selection  also  considered  work  performed  by  other  groups.  The  GPALS  CRWG 
has  examined  and  reported  on  five  related  tools.  Most  of  these  tools  are  available  on  VAX 
or  Sun  platforms.  The  Air  Force  Software  Technology  Support  Center  (STSC)  has  reported 
on  several  categories  of  software  tools,  testing  tools  being  one  of  these  categories  [Sittenau- 
er  1991].  The  role  of  the  STSC  is  to  assist  Air  Force  Software  Development  and  Support 
Activities  in  the  selection  of  technologies  that  improve  the  quality  of  Air  Force  software 
products  and  increase  the  productivity  of  its  efforts;  the  focus  is  on  the  long-term  develop¬ 
ment  and  support  of  Mission  Critical  Computer  Resources  (MCCR)  software.  STSC 
looked  at  test  tools  that  support  Ada,  assembler,  ATLAS,  C,  Fortran,  and  Jovial  program¬ 
ming  languages  running  on  DEC/VAX  equipment,  HP/Apollo  and  Sun  workstations,  or 
IBM  and  Macintosh  personal  computers  (PCs).  The  STSC  1991  report  provides  half-page 
descriptions  of  some  twenty  eight  tools,  and  tool  critiques  based  on  hands-on  application 
for  eight  of  these  tools.2  Table  4-1  identifies  the  tools  examined  in  the  CRWG  and  STSC 
studies.  Care  was  taken  not  to  duplicate  this  previous  work. 
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Table  4-1.  Tools  Examined  in  the  CRWG  and  STSC  Studies 


STUDY 

TOOL 

NAME 

TOOL  SUPPLIER 

LANGUAGES 

SUPPORTED 

TEST 

CAPABILITIES 

DC 

g 

O 

s 

** 

s 

u 

Ada 

+ 

+ 

U 

Fortran 

Other 

Test  Management 

Problem  Reporting 

Static  Analysis 

Dynamic  Analysis 

Regression  Analysis 

STSC 

Automator  qa 

Direct  Technology 

♦ 

+ 

* 

3 

El 

Software  Recording  Corporation 

a 

D 

a 

D 

D 

El 

Bloodhound 

Goldbrick  Software 

i* 

* 

+ 

- 

3 

El 

Logiscope 

Verilog,  Inc. 

u 

El 

El 

El 

El 

El 

PC  Metric 

SET  Laboratories,  Inc. 

El 

El 

El 

El 

El 

VAX  PCA 

Digital  Equipment  Corp. 

* 

* 

* 

* 

* 

El 

VAX  SCA 

Digital  Equipment  Corp. 

* 

* 

* 

* 

* 

El 

Test  Manager 

Digital  Equipment  Corp. 

+ 

+ 

+ 

*+ 

+ 

_ 

El 

CRWG 

ADAM  AT 

Dynamics  Research  Corp. 

El 

El 

Analyze 

Martin  Marietta  IS 

El 

a 

El 

SASET 

Martin  Marietta  IS 

+ 

♦ 

+ 

- 

+ 

El 

SQMS 

Martin  Marietta  IS 

El 

El 

El 

El 

El 

El 

□ 

SP/CE 

Martin  Marietta  IS 

♦ 

♦ 

_L 

3 

El 

_ 

_ 

+  -  Language  independent 
*  -  Most  VAX  supported  languages 


Tabic  4-2  identifies  the  tools  already  examined  in  the  IDA  study  and  Table  4-3  identi¬ 
fies  several  additional  tools  awaiting  examination  as  part  of  this  ongoing  work.  In  most 
cases,  this  latter  group  are  new  tools  due  to  be  released  late  in  1992  or  early  in  1993.  Some 
offer  unique  capabilities  that  fill  identified  gaps  in  testing  tool  functionality.  PARTAMOS, 
for  example,  is  expected  to  provide  for  reproducible  testing  of  concurrent  Ada  software. 
Others  provide  capabilities  that  are,  as  yet,  not  commonly  available.  For  example,  Ada- 
ASSURED  and  the  Ada  Quality  Toolset  will  check  for  conformance  of  code  with  the  Soft¬ 
ware  Productivity  Consortium  (SPC)  Ada  style  guidelines  [SPC  1991]  selected  by  SDIO. 
The  U.S.  Government  is  sponsoring  development  of  ARC  S  ADCA,  and  NATO  the  devel¬ 
opment  of  the  Test  Support  Toolset  of  the  NATO  Ada  Programming  Support  Environment 


2.  The  1992  update  of  this  report,  divided  into  two  reports  Test  Preparation,  Execution,  and  Anal¬ 
ysis  Tools  Report  [Price  1992a]  and  Source  Code  Static  Analysis  Test  Tools  Report  [Price  1992b], 
does  not  include  any  tool  critiques. 
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Table  4*2.Tools  Examined  In  the  IDA  Study 


LANGUAGES 

SUPPORTED 


TEST 

CAPABILITIES 


ADADL  Processor 


AdaQuest 


AutoFlow-Ada 


DDTs 


EDSA 


GrafBrowse 


LDRA  Testbed 


Logiscope 


MALPAS 


Metrics  Manager 


I S2E2 


QualGen 


S-TCAT 


SQArManager 


SRE  Toolkit 


SoftTest 


T-PLAN 


TBGEN 


TCAT 


TCAT-PATH 


TCMON 


TDGen 


TSCOPE 


TST 


Test/Cycle 


TestGen 


s  w  E 

g  i.  S  £ 

tiill 

u  £  O  £ 


Software  Systems  Design 


General  Research  Corp. 


AutoCASE  Technology 


QualTrak  Corp. 


Array  Systems  Computing,  Inc. 


Software  Systems  Design 


Program  Analysers,  Ltd. 


Verilog,  Inc. 


TA  Consultancy  Services,  Ltd. 


Computer  Power  Group,  Inc. 


Quality  Engineering  Software,  Inc. 


Software  Systems  Design 


Software  Research,  Inc. 


Software  Quality  Automation 


Software  Quality  Engineering 


Bender  &  Associates 


Programming  Environments,  Inc. 


Software  Quality  Assurance,  Ltd. 


TestwellOy 


Software  Research,  Inc. 


Software  Research,  Inc. 


TestwellOy 


Software  Research,  Inc. 


Software  Research,  Inc. 


STARS  Foundation  Repository 


Computer  Power  Group,  Inc. 


Software  Systems  Design 


+  -  Language  independent 
F  -  Future  capability 


(APSE).  Both  of  these  toolsets  are  expected  to  provide  a  broad  range  of  static  and  dynamic  testing  ca 
pabilities. 
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TOOL  DEVELOPER/ 
SUPPLIER 


LANGUAGES  I  TEST 
SUPPORTED  I  CAPABILITIES 


e  -S 

|  |  .2 

S  a.  * 

tt  »i  i 

W  CQ 

e  „  e 

«  S  < 

u  S  2  z 

•  «  49  £ 

-  s  £  a 

o  £  £  S 


ARC  SADCA 

Optimization  Technology,  Inc. 

KID 

U 

UUUD 

Ada-ASSURED 

GrammaTech,  Inc. 

UD 

-j 

DU 

Ada  Quality  Toolset 

Marlstone  Software  Technology,  Inc. 

EMM 

■ 

D 

DU 

Battlemap  Analysis  Tool 

McCabe  &  Associates 

an 

DU 

DUUD 

CaseQMS 

A-  V  ,  &  Computer  Systems,  Inc. 

EE 

+ 

+ 

+ 

UD 

_z 

Instrumentation  Tool  for  Ada 

McCabe  &  Associates 

EMM 

UD 

PARTAMOS 

Alcatel  Austria 

EMM 

M 

DUUD 

QES/Architect 

Quality  Engineering  Software,  Inc. 

+  + 

♦ 

+ 

DU 

DU 

QES /Programmer 

Quality  Engineering  Software,  Inc. 

+  + 

DDD 

Z 

UD 

QTET 

QualTrak  Corp. 

+  + 

+ 

+ 

DU 

DD 

DU 

QUES 

Software  Productivity  Solutions,  Inc. 

EMM 

L_ 

DU 

Quality  TEAM 

Scopus  Technologies 

+  + 

+ 

+ 

+ 

UD 

Requirements  Tracer 

Teledyne  Brown  Engineering 

+  + 

DDD  a 

HZ 

SLICE 

McCabe  &  Associates 

KIEfl 

DU 

DUUD 

START 

McCabe  &  Associates 

+  + 

+ 

+ 

r 

zz 

UD 

SQMS 

Software  Quality  Tools  Corp. 

+  + 

+ 

+ 

DUUU 

SWEEP 

Software  Productivity  Consortium 

+  + 

*+ 

+ 

DU 

zz 

+  -  Language  independent 


4.2  Method  of  Examination 


Each  static  and  dynamic  analysis  tool  was  used  to  test  several  small  Ada  programs.  The  goal  of 
these  initial  tool  applications  was  to  allow  the  examiner  to  gain  familiarity  with  overall  tool  oper¬ 
ation.  Each  tool  was  subsequently  applied  to  the  same  Ada  program.  This  software  was  the  Ada 
Lexical  Analyzer  Generator  program  that  creates  a  lexical  analyzer  or  “next-token”  procedure  for 
use  in  a  compiler  or  other  language  processing  program  [Meeson  1989].  It  was  developed  for  the 
Software  Technology  for  Adaptable,  Reliable  Systems  (STARS)  program  and  consists  of  several 
Ada  subprograms  with  a  total  of  over  three  thousand  lines  of  code.  In  the  absence  of  a  historical 
test  database,  the  test  management  and  problem  reporting  tools  were  examined  using  the  sample 
test  database  provided  by  each  supplier. 
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Generally,  suppliers  provided  their  latest  tool  release  for  the  examination.  In  a  couple 
of  cases,  only  demonstration  versions  were  available.  In  each  such  case,  however,  the  dem¬ 
onstration  version  was  fully  functional  and  only  limited  by  the  number  of  inputs  it  could 
accept. 

4.3  CASE  System  Support  for  Software  Testing 

A  recent  survey  of  CASE  vendors,  performed  on  behalf  of  the  U.S.  Air  Force,  found 
that  nearly  25%  of  the  examined  products  claim  explicit  support  for  software  testing  activ¬ 
ities  [Hook  1991].  The  goal  of  incorporating  testing  support  into  a  CASE  system  is  to  pro¬ 
vide  easy  access  to  testing  tools  and  so  facilitate  continual  evaluation  of  evolving  software. 
This  evaluation  can  be  used  to  ensure  timely  detection  of  faults  and  provide  the  software 
developer  with  feedback  to  guide  the  development  process,  thus  encouraging  a  better  inte¬ 
gration  of  testing  with  other  software  development  activities. 

Table  4-4  indicates  the  types  of  test  support  provided  by  current  CASE  systems.  At  the 
code  level,  coverage  and  performance  analysis  are  the  most  common  types  of  support  pro¬ 
vided.  These  capabilities  are  similar  to  those  provided  by  independent  test  tools  and  are 
sometimes  available  as  stand-alone  products.  However,  it  is  during  earlier  stages  of  soft¬ 
ware  development  that  CASE  systems  hold  the  most  potential  for  improving  the  integration 
of  testing  with  other  development  activities.  Several  CASE  tools  provide  requirements 
traceability,  use  simulation  and,  occasionally,  executable  specifications  to  indirectly  sup¬ 
port  testing.  Recently,  more  direct  support  in  terms  of  test  generation,  test  plan  tracking, 
and  specification  analysis  based  on  user-defined  rules  has  become  available.  Examination 
of  testing  tools  that  are  part  of  a  CASE  system  is  still  needed.  In  particular,  the  question  of 
how  to  achieve  the  necessary  integration  of  independent  and  CASE-based  testing  tools  to 
provide  a  comprehensive  automated  test  capability  must  be  addressed. 


4.4  Development  Environment  Support  for  Software  Testing 

A  previous  IDA  study  identified  twenty  eight  U.S.  companies  that  supply  validated  Ada 
compilers  [Hook  1991].  All  these  vendors  provide  a  minimum  set  of  tools  for  Ada  code  de¬ 
velopment  including  the  compiler,  editor,  debugger,  library  manager,  and  run-time  envi¬ 
ronment  The  Ada  language  definition  allows  Ada  compilers  to  provide  considerably  more 
static  analysis  than  is  possible  for  older  languages  such  as  Fortran.  Capabilities  such  as  type 
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checking  and  range  checking,  for  example,  are  always  provided.  The  other  types  of  testing 
support  provided  vary  quite  considerably.  As  shown  in  Table  4-5,  coverage  analysis,  per¬ 
formance  analysis,  and  cross-referencing  are  the  most  common  testing  capabilities  support¬ 
ed. 

Some  vendors  (DEC,  IBM,  and  Verdix)  demonstrate  a  movement  towards  providing  an 
integrated  development  environment  that  encompasses  most  phases  of  the  software  devel¬ 
opment  life  cycle.  In  this  case,  for  the  implementation  phase,  there  are  tool  sets  offered  with 
the  compiler.  For  requirements  specification  and  design,  these  development  environments 
support  various  off-the-shelf  CASE  systems. 


Table  4-4.  CASE-based  Testing  Support 
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5.  TEST  MANAGEMENT 

This  section  identifies  key  capabilities  of  the  examined  tools  in  terms  of  the  support 
provided  for  test  management.  It  is  intended  to  provide  a  quick  overview  of  the  types  of 
automated  support  available  in  each  area  and  insight  into  how  this  support  can  be  used  to 
facilitate  software  testing. 

Previously  Table  4-2  identified  six  tools  as  providing  test  management  capabilities  and 
one  other,  LDRA  Testbed,  as  currently  being  extended  to  provide  these  capabilities  in  the 
near  future.  The  functionality  of  these  tools  is  further  detailed  in  Table  5-1. 


Table  5-1.  Test  Management  Capabilities  of  Examined  Tools 


TOOL  NAME 

Test  Planning  &  Documentation 

n 

Progress 

Monitoring 

Productivity  Analysis  j 

Test  Plan  Capture 

Test  Case  Capture 

Test  Log 

Test  Inventory  Software 

Test  Inventory  Tools 

User-def.  Data  Entry  Templates 

Cross-Referencing  Test  Items 

Version  Control 

IEEE  Standards  Support 

DoD/Mil  Standards  Support 

User-defined  Reports 

Predefined  Reports 

Requirements/Test  Tracing 

Change  Analysis 

Test  Case  Status  Reporting 

Requirements  Coverage 

Test  Schedule  Reporting 

Cost  Reporting 

Reliability  Analysis 

LDRA  Testbed 

Ki 

Metrics  Manager 

El! 

QES/Manager 

El 

El 

El 

El 

SQA:Manager 

El 

El 

El 

El 

El 

El 

El 

El 

El 

El 

El 

El 

El 

El 

El 

SRE  Toolkit 

1 

11 

El 

El 

T-PLAN 

El 

El 

El 

El 

El 

EB 

El 

El 

El 

El 

El 

El 

El 

Test/Cycle 

El 

El 

El 

El 

El 

El 

El 

El 

El 

El 

El 

El 

El 

_ 

El 

Q 

F  -  Future  capability 


Two  additional  tools,  SoftTest  and  T,  are  also  discussed.  Although  not  properly  classed 
as  test  management  tools,  both  of  these  provide  some  support  for  requirements  mapping 
and  progress  monitoring. 
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5.1  Test  Planning  and  Documentation 

Test  planning  is  a  prerequisite  to  effective  management  of  test  activities.  It  provides  the 
base  against  which  required  test  activities  can  be  scheduled,  test  resources  can  be  estimated, 
and  the  progress  of  test  activities  can  be  tracked. 

Of  the  examined  tools,  QES/Manager,  SQA:Manager,  T-PLAN,  and  Test/Cycle  pro¬ 
vide  explicit  support  for  test  planning,  though  they  take  somewhat  different  approaches. 
QES/Manager  and  SQA:Manager  incorporate  a  predefined  test  model  that  defines  the  rela¬ 
tionship  among  test  objects  such  as  documents,  test  cases,  and  products.  In  the  case  of 
SQA:Manager  this  model  follows  the  Institute  for  Electrical  and  Electronics  Engineers 
(IEEE)  standard  test  model  [ANSI/IEEE  1983].  The  QES/Manager  test  model  groups  test 
cases  into  test  drivers  that  specify  an  execution  sequence  for  those  test  cases.  Test/Cycle 
defines  the  types  of  permissible  test  objects,  but  allows  the  user  to  define  the  links  between 
these.  It  is  worth  noting  that  software  builds  are  one  of  Test/Cycles  object  types,  allowing 
this  tool  to  explicitly  support  incremental  software  development.  T-PLAN  provides  the 
most  flexibility.  It  requires  a  user  to  start  by  defining  the  underlying  test  model,  although 
an  in-house  developed  test  methodology  can  be  used  as  the  source  of  the  test  model  if  de¬ 
sired. 

With  the  necessary  model  established,  these  tools  capture  similar  information  for  test 
cases  and  groupings  of  these  test  cases.  They  differ  in  the  other  types  of  information  cap¬ 
tured.  Most  significantly,  only  T-PLAN  and  Test/Cycle  explicitly  capture  requirements  and 
trace  these  to  testing  data  (see  Section  5.2),  and  only  SQA:Manager,  T-PLAN,  and  Test/ 
Cycle  explicitly  document  a  test  plan.  All  the  tools  except  QES/Manager  do,  however,  trace 
test  data  to  the  software  items  under  test  (A  capability  offered  by  QES/Manager,  unique 
among  these  tools,  is  the  ability  to  simulate  the  test  data.)  Examples  of  other  information 
that  can  be  captured  by  some  of  these  tools  include  a  test  schedule  and  an  inventory  of  test 
tools.  All  these  tools  provide  user-tailorable  templates  to  support  data  entry. 

The  tools  also  differ  in  their  reporting  on  the  contents  of  the  test  library.  QES/Manager 
requires  the  user  to  define  all  report  formats,  and  Test/Cycle  provides  a  range  of  predefined 
report  formats.  SQA:Manager  and  T-PLAN  support  both  predefined  and  user-defined  re¬ 
port  formats.  In  addition  to  the  available  IEEE  standards,  SQA:Manager  supports  applica¬ 
ble  DoD  standards  DoD-STD-2167A  and  DoD-STD-2168,  and  military  standard  MIL- 
STD-480. 
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5.2  Requirements  Mapping 

The  ability  to  trace  the  relationship  between  software  requirements  and  test  items  pro¬ 
vides  valuable  insight  into  the  completeness  and  effectiveness  of  both  test  planning  and  test 
execution.  It  is  also  a  prerequisite  for  the  change  analysis  that  determines  the  potential 
scope  of  effect  of  a  proposed  requirements  change.  The  ability  to  provide  this  support  is 
one  of  the  major  differences  between  the  tools  in  this  category.  It  is  available  with  T-PLAN 
and  Test/Cycle. 

Test/Cycle  uses  requirements  validation  matrices  to  cross-reference  requirements 
against  software  builds,  test  runs,  and  test  cases.  These  matrices  can  be  examined  to  ensure 
that  all  requirements  are  appropriately  covered,  providing  quick  insight  into  test  planning 
completeness.  T-PLAN  links  requirements  to  test  cases  via  test  conditions  that  can  be 
grouped  to  reflect,  for  example,  valid/invalid  categories,  system  releases  or  versions.  It  also 
reports  on  the  test  items  affected  by  a  change  to  a  test  requirement,  in  addition  to  the  change 
analysis  provided  for  other  types  of  test  items. 

SoftTest  and  T  provide  requirements  traceability  in  a  different  way.  Here  a  require¬ 
ments  specification  is  used  to  guide  the  generation  of  test  cases.  Hence,  test  cases  are  auto¬ 
matically  linked  to  defined  functional  requirements.  Both  tools  provide  matrices  that  give 
a  quick  visual  guide  to  the  cross-referencing  between  functional  requirements  and  test  cas¬ 
es. 


5.3  Test  Progress  Monitoring 

Test  progress  monitoring  is  important  for  effective  management  of  test  activities.  By 
tracking  actual  progress  against  planned  progress,  managers  can  get  an  early  indication  of 
potential  schedule  slips  to  support  timely  decision  making.  The  early  identification  of  qual¬ 
ity  shortfalls  is  another  piece  of  valuable  information.  The  data  collected  during  test 
progress  monitoring  can  also  be  used  to  assess  various  overall  software  development  indi¬ 
cators  and  quality  indicators  (see,  for  example,  [AFSCP  86,  AFSCP  87]).  Progress  moni¬ 
toring  is  largely  based  on  a  log  of  testing  activities.  Data  is  entered  into  the  test  log 
manually  or,  in  some  cases,  can  be  imported  from  a  test  execution  tool. 

SQA:Manager  and  T-PLAN  capture  similar  information  for  the  test  log.  Using  this  in¬ 
formation,  SQA:Manager  reports  on  the  status  of  each  test  case,  that  is,  the  number  of  tests 
passed,  failed,  and  aborted,  and  the  number  of  incidents  raised.  T-PLAN  reports  whether 
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each  test  procedure  has  been  tested,  date  of  last  test,  and  whether  a  re-test  is  required,  to¬ 
gether  with  details  on  the  conduct  of  the  individual  tests  performed.  Using  the  schedule  in¬ 
formation  entered  for  each  test  specification,  T-PLAN  also  compares  estimated  and  actual 
levels  of  effort  to  determine  the  outstanding  effort  and  report  on  the  percentage  completion. 
This  reporting  is  available  for  test  planning,  testing,  regression  testing  and  review  activities. 

Test/Cycle  reports  on  the  validation  status  of  requirements,  builds,  and  test  runs  in 
terms  of  the  percentage  of  test  cases  passed.  It  provides  this  for  each  leaf  requirement  or 
requirement  subtree  in  its  requirements  hierarchy.  Additional  reports  summarize  the  overall 
status  of  requirements,  builds,  and  test  cases,  whereas  a  test  log  report  provides  detailed  in¬ 
formation  on  the  status  of  individual  test  cases. 

SoftTest  and  T  report  on  the  requirements  coverage  achieved  through  testing  to  date. 
SoftTest  reports  requirements  coverage  in  terms  of  the  number  of  functional  variations  test¬ 
ed  with  respect  to  those  testable;  this  requires  the  user  to  manually  enter  the  results  of  test 
case  execution.  T  also  maps  user-supplied  test  results  to  requirements  to  report  on  test  ad¬ 
equacy  with  respect  to  requirements  coverage.  It  provides  a  test  comprehensiveness  mea¬ 
sure  that,  at  the  user’s  choice,  combines  requirements,  input/output,  and  structural  cover¬ 
age. 

Reliability  analysis  is  also  used  to  monitor  test  progress  against  a  stated  objective.  A 
failure  intensity  objective,  for  example,  specifies  the  expected  number  of  software  failures 
per  unit  of  time,  whereas  a  reliability  objective  specifies  the  probability  of  failure-free  op¬ 
eration.  By  looking  at  the  occurrence  of  software  failures  during  testing  activities,  it  is  pos¬ 
sible  to  estimate  the  number  of  defects  remaining  in  a  piece  of  software  and  determine  (with 
confidence  intervals)  the  additional  time  or  resources  needed  to  reach  the  goal  objective. 
By  predicting  the  reliability  of  software  after  modification,  these  measures  can  also  help  to 
time  the  performance  of  maintenance  activities,  for  example,  the  addition  of  new  features. 
Under  the  proper  conditions,  reliability  measures  can  be  used  to  help  determine  the  effec¬ 
tiveness  of  particular  software  engineering  practices,  or  the  effects  of  process  improve¬ 
ments. 

Many  different  reliability  models  have  been  proposed.  The  two  most  common  are  Mu¬ 
sa’s  basic  execution-time  model  and  the  Musa-Okumoto  logarithmic  Poisson  execution¬ 
time  model  [Musa  1987].  Both  models  characterizes  failures  as  a  nonhomogeneous  Poisson 
distribution.  SRE  Toolkit  supports  reliability  analysis  using  both  of  Musa’s  models,  where¬ 
as  SQA:Manager  uses  the  Musa-Okumoto  model.  Both  tools  provide  failure  intensity  and 
reliability  reports  that  include  the  amount  of  additional  testing  time  needed  to  meet  a  tar- 
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geted  reliability,  and  an  estimation  of  how  many  more  problems  are  likely  to  be  found  dur¬ 
ing  that  additional  testing.  They  both  support  cost  analysis.  SQA:Manager  relates  the  hours 
spent  in  test  activities  and  in  problem  resolution  to  cost  rates  in  a  cost  base  to  report  the  cost 
of  these  activities.  Using  data  on  the  cost  of  failure  identification  and  correction,  and  the 
cost  of  operational  failure,  SRE  Toolkit  maps  total  life  cycle,  system  test,  and  operational 
life  costs  against  a  specified  failure  intensity  objective. 

SRE  Toolkit  supports  a  number  of  additional  features.  For  example,  the  user  can  spec¬ 
ify  a  failure  time  adjustment  to  take  account  of  incremental  delivery  of  software  to  the  sys¬ 
tem  test  process  and  a  testing  compression  factor  to  specify  the  ratio  of  field  to  test 
execution  time.  The  toolkit  can  be  instructed  to  interpret  individual  failure  entries  as  inde¬ 
pendent  failure  events  or  to  perform  grouped  data  analysis.  A  suite  of  prototype  programs 
provides  further  information  such  as  summary  statistics  for  each  recording  period,  esti¬ 
mates  of  resource  usage  calendar  time  parameters  from  resource  usage  data,  and  plots  of 
completion  date  for  testing  and  life  cycle  costs  versus  failure  intensity  objective. 

In  addition  to  that  discussed  here,  information  on  the  status  of  identified  problems  (see 
Section  6)  and  the  coverage  achieved  during  dynamic  testing  (see  Section  8.2)  also  provide 
insight  into  the  status  of  testing  activities. 


5.4  Productivity  Analysis 

Productivity  data,  like  quality  data,  can  be  used  to  monitor  the  efficiency  of  the  software 
development  process.  It  supports  the  identification  of  those  instances  where  process  im¬ 
provements  are  needed,  and  the  effectiveness  of  process  changes.  While  several  tools  sup¬ 
port  the  collection  and  analysis  of  quality  data.  Metrics  Manager  is  the  only  examined  tool 
that  provides  productivity  analysis.  As  such,  it  looks  at  a  user-defined  Management  Infor¬ 
mation  System  (MIS)  function,  collecting  data  on  a  monthly,  quarterly,  or  annual  basis  to 
monitor  the  performance  of  the  organization  and  track  the  impact  of  new  methods,  organi¬ 
zational  structures,  and  technologies.  Metrics  Manager  is  supported  by  an  industry  database 
that  allows  comparison  of  organizational  data  against  industry  statistics. 
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6.  PROBLEM  REPORTING 

The  primary  purpose  behind  problem  reporting  is  to  ensure  that  all  identified  problems 
are  addressed.  The  data  inherent  in  this  activity  serves  several  additional  purposes.  It  pro¬ 
vides  a  valuable  insight  into  both  the  software  status  and  the  progress  of  development  and 
test  activities.  Additionally,  it  provides  much  of  the  data  needed  to  drive  continuous  process 
improvement  activities. 

Four  tools  that  support  problem  reporting  were  examined.  One  of  these,  DDTs,  address¬ 
es  this  function  exclusively.  For  SQ  A -.Manager,  T-PLAN,  and  Test/Cycle,  problem  report¬ 
ing  is  only  one  of  the  types  of  support  provided  for  software  testing.  Consequently,  it  is  not 
surprising  that  there  are  several  significant  differences  between  these  two  types  of  support. 
The  capabilities  of  the  tools  are  summarized  in  Table  6- 1 . 


Table  6-1.  Problem  Reporting  Capabilities  of  Examined  Tools 


TOOL 

NAME 

Report 

Types 

Details  Captured 

Import  Capability  | 

Reporting 

Stnd. 

Con. 

Distrib. 

ArchiL 

|  Defect  Reports 

|  Change  Requests 

|  Incident  Reports 

Define  Categories 

Set  Status 

Assign  Resolution 

1 

3 

1 

as 

& 

Set  Effort 

#  Priority/Severity  Levels 

Set  Phase  Introduced 

Set  Changed  Software 

Predefined  Reports 

User-defined  Reports 

Report  Filters 

£* 

as 

Z 

m 

n. 

U 

t 

1 

t 

3 

o 

IEEE  Standards 

DoD  Standards 

Automatic  Notification 

Multiple  Projects 

Administrative  Functions 

DDTs 

El 

El 

El 

El 

El 

El 

El 

El 

a 

El 

El 

El 

El 

El 

El 

El 

El 

El 

El 

El 

El 

SQArManager 

El 

■ 

El 

El 

El 

El 

El 

El 

a 

El 

El 

El 

El 

El 

T-PLAN 

HI 

■ 

El 

El 

■ 

El 

El 

m 

El 

□ 

Test/Cycle 

El 

El 

El 

El 

El 

□ 

El 

n 

El 

El 

d 

6.1  Report  Types  and  Details  Captured 

The  ability  to  distinguish  among  different  types  of  problems,  and  perform  separate 
tracking  and  reporting  for  each  type,  is  very  useful  in  monitoring  the  software  development 
progress  and  planning  further  development  activities.  The  common  types  of  problem  re¬ 
ports  are  incident  reports,  defect  reports,  and  change  requests.  Only  Test/Cycle  has  all  these 
problem  types,  collectively  called  work  requests ,  built  in.  Although  in  its  basic  form  DDTs 
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only  distinguishes  between  defects  and  change  requests,  it  can  be  customized  to  also  accept 
incident  reports.  SQA:Manager  distinguishes  between  incidents  and  defects.  T-PLAN 
tracks  and  reports  a  single  problem  type,  called  service  queries. 

By  and  large,  all  the  tools  capture  similar  details  about  identified  problems.  Data  entry 
is  guided  by  user-tailorable  templates.  DDTs  allows  the  provision  of  supplemental  infor¬ 
mation  that  is  kept  separately,  but  linked  to  a  defect  report  This  additional  information  can 
be  used,  for  example,  to  include  the  data  files  needed  to  reproduce  a  problem.  The  test 
item(s)  to  which  problem  reports  are  linked  affects  the  type  of  tracking  that  can  be  per¬ 
formed.  SQArManager  and  T-PLAN  link  them  to,  respectively,  test  cases  and  test  specifi¬ 
cations.  DDTs  and  Test/Cycle  link  problem  reports  to  software  items. 

DDTs  provides  a  good  example  of  the  additional  power  provided  by  tools  that  focus  ex¬ 
clusively  on  problem  reporting.  Here  problems  have  a  specified  life  cycle  defined  in  terms 
of  states  and  state  transitions.  The  system  administrator  is  permitted  to  modify  this  life  cy¬ 
cle. 


6.2  Import  Capability 

A  flexible  import  facility  is  a  valuable  feature.  It  allows  data  generated  using  other 
tools  to  be  incorporated  in  a  common  problem  database.  This  is  useful,  for  example,  when 
different  problem  reporting  tools  are  used,  perhaps  to  cater  for  different  development  orga¬ 
nizations  or  host  machines.  DDTs  and  SQA:Manager  provide  this  capability. 

6.3  Reporting  Capabilities 

T-PLAN,  Test/Cycle,  and  DDTs  provide  predefined  report  formats.  In  the  case  of 
DDTs,  these  reports  conform  to  DoD-STD-2167A  and  the  proposed  EEEE  standard  classi¬ 
fication  for  software  errors,  faults,  and  failures  [IEEE  1987].  For  T-PLAN  the  available  sta¬ 
tistical  reports  analyze  the  total  numbers  of  defects,  or  queries,  by  classification.  Frequency 
of  defects  and  defect  resolution  is  also  provided,  as  well  as  the  percentage  complete  and 
outstanding  effort  required  to  complete  approved  changes.  Test/Cycle  reports  provide  only 
work  request  descriptions  and  a  work  request  log.  DDTs  also  allows  a  user  to  define  his 
own  report  formats,  as  does  SQA:Manager.  In  these  cases,  a  number  of  predefined  report 
filters  and  sorting  keys  are  provided  to  support  reporting  based  on  any  problem  character- 
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SQA:Manager  and  Test/Cycle  report  on  the  costs  associated  with  defect  detection  and 
correction.  Of  the  examined  tools,  only  DDTs  provides  a  capability  for  automatic  weekly 
reporting  on  problems. 

In  addition  to  its  reporting  facilities,  DDTs  provides  advanced  search  and  query  func¬ 
tions. 


6.4  Standards  Conformance 

SDIO  requires  the  reporting  and  tracking  of  identified  problems  but  does  not  specify 
how  this  requirement  should  be  met  Some  additional  guidance  is  given  in  DoD-STD- 
2167A.  This  standard  requires,  for  example,  that  problems  are  classified  by  category  (soft¬ 
ware,  documentation,  or  design  problem)  and  are  assigned  one  of  five  levels  of  priority.  It 
also  requires  analysis  of  defect  trends  and  the  identification  of  any  additional  problems  in¬ 
troduced  by  a  problem  fix. 

The  examined  tools  vary  in  their  ability  to  meet  these  requirements.  As  shown  in  Table 
6-1,  only  DDTs  and  SQA.Manager  provide  five  priority  levels  as  a  default  option,  al¬ 
though,  for  the  other  tools,  the  user  can  generally  modify  the  input  template  to  allow  a  dif¬ 
ferent  set  of  levels.  The  ability  to  record  problem  classifications  is  highly  variable. 
SQA:Manager  and  T-PLAN,  for  example,  allow  user-defined  categories,  whereas  DDTs 
accepts  free-form  text  for  this  information.  None  of  the  tools  provides  explicit  support  for 
recording  the  introduction  of  new  problems  as  a  results  of  a  problem  fix.  Several  pieces  of 
information  can  support  the  analysis  of  defect  trends.  Problem  classification  and  details  on 
when  a  problem  was  inserted,  detected,  and  the  first  opportunity  for  its  detection,  for  ex¬ 
ample,  are  all  useful.  DDTs,  SQA:Manager,  and  Test/Cycle  capture  at  least  part  of  this  in¬ 
formation. 

6.5  Distributed  Architecture 

The  dedicated  problem  reporting  tool,  DDTs,  is  network  based.  This  tool  uses  electron¬ 
ic  mail  to  provide  automatic  notification  of  changes  in  problem  status  and  to  support  remote 
problem  entry.  It  also  supports  multiple  projects.  DDTs  also  provides  access  controls  and 
various  other  administrative  capabilities.  These  additional  capabilities  range  from  checking 
and  repairing  the  database  to  template  definition. 
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7.  STATIC  ANALYSIS 

Static  analysis  is  used  to  determine  the  presence  or  absence  of  particular,  limited  classes 
of  errors,  to  produce  certain  kinds  of  software  documentation,  and  to  assess  various  charac¬ 
teristics  of  software  quality.  Unlike  dynamic  analysis,  static  analysis  can  sometimes  be  per¬ 
formed  on  incomplete  or  partly  development  products  and  does  not  necessitate  costly  test 
environments.  It  cannot,  however,  replace  dynamic  analysis,  although  it  can  be  used  to 
guide  and  focus  dynamic  testing.  Previously  Table  4-2  identified  fourteen  tools  as  support¬ 
ing  static  analysis.  The  functions  provided  by  these  tools  are  summarized  in  Table  7-1. 


Table  7-1.  Static  Analysis  Capabilities  of  Examined  Tools 


TOOL  NAME 

Complexity  Analysis  | 

Data  Flow  Analysis  | 

Control  Flow 
Analysis 

Information  Flow  Analysis  | 

Standards  Conf.  Analysis  j 

Quality  Analysis  | 

Cross-Reference  Analysis  | 

Browsing  j 

Symbolic  Evaluation  | 

Specification  Compliance  j 

Pretty  Printing  | 

Structure  Analysis 

Path  Analysis 

Code  Statistics 

Flowgraph  Generation 

Call  Graph  Generation 

u 

El 

El 

AdaQuest 

El 

F 

B 

AutoFlow-Ada 

El 

B 

El 

EDSA 

El 

El 

El 

El 

El 

GrafBrowse 

El 

El 

ZJ 

LDRA  Testbed 

El 

El 

El 

El 

El 

El 

El 

El 

El 

El 

El 

Logiscope 

El 

El 

El 

El 

El 

El 

rz 

El 

El 

MALPAS 

El 

El 

El 

El 

El 

El 

El 

El 

QualGen 

El 

S-TCAT 

El 

El 

El 

TCAT 

El 

El 

El 

TCAT-PATH 

El 

El 

El 

El 

El 

TST 

_ 

El 

El 

El 

TestGen 

El 

El 

El 

_ 

_ 

□ 

F  -  Future  capability 


7.1  Complexity  Analysis 

Complexity  measures  are  put  to  various  test-related  uses.  McCabe  has  developed  a 
method,  called  Structured  Testing,  that  uses  cyclomatic  complexity  to  guide  the  selection 
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of  a  minimum  set  of  required  paths  to  test  [McCabe  1982].  Complexity  measures  are  also 
used  to  estimate  the  number  of  defects  present  in  a  piece  of  software  and  to  identify  pieces 
of  code  that  are  potentially  defective. 

Models  for  estimating  program  complexity  have  been  based  on  various  characteristics 
of  software  structure  and  semantics.  The  best-known  set  of  complexity  measures  are  all  ap¬ 
plied  at  the  program  unit  level.  They  are  McCabe’s  cyclomatic  complexity  metrics  [McCabe 
1976]  and  Halstead’s  software  science  metrics  [Halstead  1977].  Whereas  cyclomatic  com¬ 
plexity  is  control  oriented,  the  Halstead  metrics  are  text  oriented.  As  well  as  variations  on 
each  of  these  measures,  there  are  many  other  program-level  measures.  In  contrast,  relatively 
few  measures  for  assessing  design-level  complexity  have  been  proposed.  Perhaps  the  most 
common  design-level  measures  are  those  developed  by  Mohanty  that  are  based  on  a  call 
graph  [Mohanty  1976],  and  basic  subtrees,  a  variation  on  cyclomatic  complexity.  Measures 
for  assessing  requirements  complexity  are  similarly  scarce  and  not  supported  by  any  of  the 
examined  tools.  Table  7-2  identifies  the  different  types  of  complexity  measures  that  are  pro¬ 
vided. 


Table  7-2.  Supported  Complexity  Measures 


TOOL  NAME 

Unit  Level 

I®  teg 
Level 

Basic  Blocks  (#) 

Basic  Block  (Avg.  Len.) 

Cyclomatic  Complexity 

Essential  Complexity 

Intervals  (Ord.  1) 

Intervals  (Max.  Ord.) 

Control  Flow  Knots 

Essential  Knots 

Paths  (#) 

Paths  (Avg.  Length) 

Paths  (Max/Min) 

Iteration  Groups  (Max.) 

Halstead’s  Metrics 

Basic  Subtrees  (SI) 

Mohanty’s  Metrics 

ADADL  Processor 

LDRA  Testbed 

T 

T 

~r 

1~ 

~T~ 

1“ 

T" 

1~ 

T" 

Logiscope 

'l 

T 

T 

MALPAS 

~r 

TCAT-PATH 

T~ 

T~ 

~r 

~r 

T~ 

-J 

n 

TestGen 

m 

^ i — 

1~ 
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_ 
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Twenty  years  of  theoretical  and  empirical  evaluations  have  failed  to  produce  consistent, 
hard  evidence  of  the  accuracy  of  particular  measures  or  on  the  respective  value  of  alterna¬ 
tive  measures.  Consequently,  these  measures  should  be  used  as  indicators,  rather  than  ab¬ 
solute  measures  of  software  properties. 


7.2  Data  Flow  Analysis 

Data  flow  analysis  is  based  on  consideration  of  the  sequences  of  events  that  occur  along 
the  various  paths  through  a  program.  It  is  used  to  detect  data  flow  anomalies,  of  which  three 
types  are  commonly  recognized:  (1)  a  variable  whose  value  is  undefined  is  referenced,  (2) 
a  defined  variable  is  redefined  before  it  is  referenced,  or  (3)  a  defined  variable  is  undefined 
before  it  is  referenced.  While  the  first  of  these  indicates  an  actual  program  defect,  the  latter 
two  types  of  anomaly  may  indicate  questionable  variable  usage  rather  than  specific  defects. 
Since  the  analysis  technique  assumes  that  all  paths  through  the  program  are  feasible,  some 
reported  anomalies  may  be  superfluous.  Data  flow  analysis  also  can  be  used  to  categorize 
procedure  parameters  as  referenced  only,  defined  only,  both  defined  and  referenced,  or  not 
used. 

LDRA  Testbed,  MALPAS,  and  EDSA  support  static  data  flow  analysis.  LDRA  Test¬ 
bed  performs  weak  data  flow  analysis  to  identify  data  flow  anomalies  of  the  types  men¬ 
tioned  above.  It  also  analyzes  procedures  calls  across  procedure  boundaries  to  report  on 
procedure  parameter  usage.  MALPAS  refines  the  classification  of  data  flow  anomalies.  For 
example,  a  data  variable  that  is  redefined  before  it  is  referenced  may  be  classified  as  either 
an  instance  where  data  is  written  twice  without  an  intervening  read,  or  as  data  being  written 
with  no  subsequent  access  on  a  given  path.  Given  a  list  of  procedure  input  and  output  pa¬ 
rameters,  MALPAS  compares  these  with  the  classes  of  data  to  produce  a  table  of  possible 
errors.  EDSA  uses  interactive  data  flow  analysis  to  facilitate  program  browsing. 


7.3  Control  Flow  Analysis 

Control  flow  analysis  is  a  process  of  examining  a  program  structure  and  identifying  ma¬ 
jor  features  such  as  entry  and  exit  points,  loops,  unreachable  code,  and  paths  through  a  pro¬ 
gram.  This  information  can  be  used  to  determine  program  complexity  and  to  aid  in  planning 
a  dynamic  test  strategy.  It  can  help  to  decide  on  strategies  for  further  analysis,  for  example, 
to  identify  where  it  might  be  beneficial  to  partition  the  code  to  reduce  the  number  of  paths 
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and,  hence,  facilitate  semantic  analysis.  The  results  of  control  flow  analysis  can  also  be 
used  to  prepare  a  diagrammatic  representation  of  the  program  structure  that  can  aid  a  user 
in  documenting  and  understanding  a  piece  of  software. 

Control  flow  analysis  is  provided  by  the  majority  of  tools  that  support  static  analysis. 
MALPAS,  TestGen,  LDRA  Testbed,  and  TCAT  family  all  report  on  unreachable  paths. 
These  may  be  generated  as  a  result  of  program  syntax,  for  example,  as  a  result  of  end  if 
statements,  or  the  position  of  a  return  statement  Even  though  they  do  not  necessarily  imply 
an  defect,  the  occurrence  of  unreachable  paths  should  be  checked.  Some  of  the  examined 
tools  go  farther.  LDRA  Testbed,  for  example,  also  reports  on  unreachable  branches  and 
other  structural  units. 

Several  of  the  tools  use  control  flow  analysis  to  generate  a  graphical  representation  of 
a  program’s  structure  as  a  logical  flow  chart  or  directed  graph.  This  allows  visual  inspection 
of  program  structure  and  complexity,  and  can  facilitate  program  understanding  at  the  unit 
level.  AutoFlow-Ada,  LDRA  Testbed,  Logiscope,  TCAT,  and  TCAT-PATH  all  generate 
fairly  sophisticated  graphical  representations  of  a  program’s  structure.  AutoFlow-Ada,  in 
particular,  provides  a  user  with  considerable  flexibility  in  generating  a  high-quality  graph¬ 
ical  flow  chart.  TestGen  uses  textual  facilities  to  produce  a  more  primitive  graph  represen¬ 
tation.  Although  MALPAS  does  not  directly  produce  a  directed  graph,  its  list  of  nodes,  with 
identification  of  successor  and  predecessor  nodes,  helps  a  user  to  draw  this  graph.  Graphi¬ 
cal  representation  of  the  calling  relationship  between  program  units  also  facilitates  program 
understanding.  GrafBrowse,  LDRA  Testbed,  Logiscope,  and  S-TCAT  generate  call  graphs 
or  call  trees. 

The  identification  of  paths  through  a  program  is  useful  for  estimating  the  resources 
needed  for  dynamic  analysis  and  then  guiding  this  testing.  AdaQuest,  LDRA  Testbed,  Lo¬ 
giscope,  MALPAS,  TCAT-PATH,  TST,  and  TestGen  all  provide  this  capability.  Even 
more  useful,  LDRA  Testbed,  Logiscope,  and  TestGen  explicidy  identify  the  values  of  log¬ 
ical  conditions  necessary  to  cause  particular  paths  to  be  followed.  Logiscope,  TCAT, 
TCAT-PATH,  and  S-TCAT  report  on  various  code  statistics.  These  statistics  range  from 
measures  such  as  the  number  of  each  type  of  operator  and  operand  occurring  in  the  soft¬ 
ware,  to  measures  of  the  average,  minimum,  and  maximum  path  length.  EDSA  provides 
interactive  control  flow  analysis  to  facilitate  browsing  along  program  paths. 

MALPAS,  LDRA  Testbed,  and  Logiscope  perform  structure  analysis  to  verify  a  pro¬ 
gram’s  conformance  to  the  principles  of  structured  programming.  Here  LDRA  Testbed 
matches  templates  of  acceptable  structures  with  the  directed  graph  of  a  program  on  a  mod- 
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ule  by  module  basis.  Matching  structures  are  successively  collapsed  to  a  single  noae  until 
either  a  single  node  is  left,  indicating  a  structured  program,  or  an  irreducible  state,  indicat¬ 
ing  an  unstructured  program.  MALPAS  and  Logiscope  perform  a  similar  reduction  to  eval¬ 
uate  the  structure. 


7.4  Information  Flow  Analysis 

Information  flow  analysis  is  used  to  examine  program  variable  interdependencies.  This 
helps  to  isolate  inadvertent  or  unwanted  dependencies,  to  indicate  how  a  program  can  be 
broken  down  into  subprograms,  and  to  identify  the  scope  of  program  changes.  For  security 
applications,  it  can  be  used  to  aid  the  identification  of  spurious  or  unknown  code.  Addition¬ 
ally,  it  supports  dynamic  testing  by  identifying  which  inputs  need  to  be  exercised  to  affect 
which  outputs. 

Both  LDRA  Testbed  and  MALPAS  provide  this  capability.  Currently  LDRA  Testbed 
is  limited  to  identifying  backward  dependencies  on  a  procedure  by  procedure  basis  and 
characterizes  variables  as  strongly  or  weakly  dependent.  Future  versions  of  LDRA  Testbed 
will  include  forward  dependencies  to  identify  variables  that  can  be  affected  by  a  particular 
input  variable.  It  will  also  support  information  flow  dependence  assertions  to  allow  com¬ 
parison  of  expected  dependencies  with  actual  dependencies. 

MALPAS  identifies  all  of  a  program’s  inputs  and  examines  each  executable  path  to 
identify  dependencies  for  each  output  variable.  These  dependencies  include  the  input  vari¬ 
ables,  constants,  and  conditional  statements  on  which  it  depends.  It  reports  on  program  unit 
inputs  and  outputs,  which  may  be  more  than  those  passed  as  parameters.  MALPAS  also 
identifies  redundant  statements. 


7.5  Standards  Conformance  Analysis 

Auditors  are  used  to  check  the  conformance  of  a  program  to  a  set  of  standards.  For  SDI 
software,  the  SPC  Ada  Quality  and  Style:  Guidelines  for  Professional  Programmers  [SPC 
1991]  defines  the  required  standards.  Although  none  of  the  tools  reported  here  supports 
these  guidelines,  ADAMAT  discussed  in  the  CRWG  study  does.  Two  new  tools,  Ada-AS- 
SURED  and  the  Ada  Quality  Toolset,  are  advertised  as  providing  this  support. 
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LDRA  Testbed  checks  conformance  to  a  set  of  standards  that  are  of  interest  to  the  pro¬ 
gramming  community;  this  includes  much  of  the  Safe  Ada  Subset.  Individual  standards  can 
be  disabled  and  the  user  can  weight  particular  standards  or  specify  acceptance  limits,  where 
appropriate.  TST  reports  on  conformance  to  a  set  of  portability  standards. 


7.6  Quality  Analysis 

As  already  mentioned,  several  tools  report  on  particular  quality  characteristics  such  as 
complexity  and  compliance  with  standards.  There  are,  however,  many  other  quality  char¬ 
acteristics  that  provide  insight  into,  for  example,  code  maintainability  and  portability. 

One  of  the  examined  tools,  Logiscope,  employs  the  Rome  Air  Development  Center 
(RADC)  quality  metrics  model  to  allow  user-defined  quality  measurement  at  three  levels 
of  abstraction  [RADC  1983].  At  the  lowest  level  of  the  model,  the  user  can  defined  upper 
and  lower  bounds  for  a  predefined  set  of  primitive  metrics.  Logiscope  distinguishes  be¬ 
tween  unit-level  metrics  and  architectural  metrics,  reporting  on  both.  The  user  can  then 
specify  algorithms  to  weight  and  combine  the  primitive  metrics  into  composite  metrics. 
These  composite  metrics  are,  in  turn,  used  to  define  quality  criteria  that  allow  classifying 
components  as,  for  example,  accepted  or  rejected,  based  on  their  computed  quality  values. 

QualGen  analyses  both  design  and  code  complexity  and  currently  interfaces  with  Lotus 
1  -2-3  for  quality  reporting.  It  provides  some  200  primitive  metrics  which,  via  Lotus,  can  be 
combined  into  user-defined  higher  level  measures.  Software  Systems  Design,  the  develop¬ 
er  of  QualGen,  is  currently  mapping  the  correspondence  of  QualGen  metrics  to  the  SPC 
Ada  style  guide. 


7.7  Cross-Reference  Analysis 

The  information  acquired  from  cross-referencing  program  entities  serves  many  purpos¬ 
es.  Perhaps  one  of  the  most  important  of  these  is  identifying  the  scope  of  a  program  change 
or  aiding  in  the  diagnosis  of  a  software  failure. 

The  ADADL  Processor  provides  extensive  cross-referencing  capabilities.  It  reports  on 
the  cross-referencing  between  program  units,  objects,  and  types.  It  also  reports  on  the  oc¬ 
currence  of  with  and  pragma  statements;  the  occurrence  of  interrupts,  exceptions,  and  ge¬ 
neric  instantiations;  and  the  usage  of  program  unit  renaming.  LDRA  Testbed  cross- 
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references  all  data  items  and  classifies  them  as  global,  local,  or  parameter  and  also  cross- 
references  procedure  usage.  Through  its  browsing  capabilities,  EDSA  provides  interactive 
cross-referencing  of  data  items  and  Ada  objects. 


7.8  Browsing 

A  browser  facilitates  program  understanding  by  allowing  the  user  to  create  and  present 
different  views  of  the  software.  This  may  include  views  that  show  the  same  piece  of  soft¬ 
ware  at  different  stages  of  development  and  views  that  omit  some  information  in  order  to 
focus  on  other  details.  A  browser  also  may  provide  the  user  with  the  ability  to  follow  the 
control  flow  or  data  flow  in  browsing  through  code.  These  capabilities  may  be  used  for  sev¬ 
eral  purposes,  for  example,  to  aid  in  reviewing  a  program  or  in  diagnosing  the  cause  of  a 
software  failure. 

EDSA  focuses  on  browsing  source  code  at  the  unit  level;  it  allows  browsing  forward  or 
backward  via  data  flow  or  control  flow.  The  user  can  construct  views  that  suppress  or  omit 
irrelevant  code  details  to  help  him  to  focus  on  the  concern  at  hand.  Special  annotations  are 
available  to  keep  track  of  the  progress  of  formal  code  verification.  GrafBrowse  chiefly  op¬ 
erates  at  the  integration  level.  Here  the  user  can  move  through  graphical  invocation  hierar¬ 
chies  (or  declaration  or  call-by  hierarchies),  pulling  up  the  relevant  pieces  of  code  as 
required.  The  TCAT  family  of  coverage  analyzers  also  allows  moving  between  graphical 
depictions  of  program  and  module  structure  and  the  associated  source  code. 

Although  not  examined  in  the  course  of  this  work,  the  new  version  of  Logiscope  also 
supports  source  code  browsing. 


7.9  Symbolic  Evaluation 

This  type  of  static  semantic  analysis  provides  a  more  complete  examination  of  a  pro¬ 
gram’s  operation.  Instead  of  actual  input  data,  symbols  such  as  variable  names  are  used  to 
simulate  program  execution.  This  allows  the  reporting  of  the  mathematical  relationships 
between  inputs  and  outputs  for  each  semantically  possible  path.  It  has  three  primary  uses. 
The  relationships  can  be  compared  against  a  program  specification  to  check  for  consisten¬ 
cy.  The  identified  path  condition,  together  with  the  expression  detailing  the  set  or  range  of 
input  data  which  causes  this  path  to  be  executed,  supports  test  data  generation.  Finally,  the 
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relationships  can  aid  in  determination  of  the  expected  output  for  a  set  of  test  data.  Only 
MALPAS  provides  this  very  useful  capability. 


7.10  Specification  Compliance  Analysis 

Specification  compliance  analysis  takes  semantic  analysis  a  step  further  by  automati¬ 
cally  comparing  a  program  against  its  formal  specification  to  identify  deviations.  This  type 
of  analysis  is  very  powerful,  but  requires  additional  work  on  the  behalf  of  the  user. 

Here  again,  MALPAS  was  the  only  examined  tool  that  provides  this  capability.  It  re¬ 
quires  program  specification  details  to  be  embedded  in  its  intermediate  language.  (These 
details  may  already  be  available  if  a  formal  specification  language  such  as  Z,  VDM,  or  OBJ 
is  being  used  in  the  development  effort)  The  output  of  the  compliance  analyzer  is  a  set  of 
threat  statements  that,  if  the  program  does  not  meet  the  specification,  presents  the  relation¬ 
ships  between  inputs  that  cause  a  deviation  to  occur. 

7.11  Pretty  Printing 

A  useful  documentation  capability,  pretty  printing  is  provided  by  the  ADADL  Proces¬ 
sor,  AutoFlow-Ada,  EDSA,  LDRA  Testbed,  and  TST. 
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8.  DYNAMIC  ANALYSIS 

This  section  reports  on  the  capabilities  provided  by  the  examined  tools  for  dynamic 
analysis  where  software  is  evaluated  based  on  its  behavior  during  execution.  Dynamic  anal¬ 
ysis  is  the  primary  method  for  validating  and  verifying  software.  Additionally,  it  is  the 
source  of  much  of  the  information  used  in  monitoring  testing  progress  and  software  quality. 
Traditionally  an  unstructured  and  labor-intensive  activity,  dynamic  analysis  is  a  significant 
cost  driver.  This  study  examined  the  dynamic  analysis  capabilities  of  fourteen  tools.  Table 
8- 1  identifies  the  particular  functionality  provided  by  each. 


Table  8-1.  Dynamic  Analysis  Capabilities  of  Examined  Tools 


TOOL  NAME 

Assertion  Analysis  j 

Coverage 
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1.  Used  in  conjunction  with  TCAT,  TCAT-PATH,  or  S-TCAT  to  animate  coverage  results. 

F  -  Future  capability 


8.1  Assertion  Analysis 

An  assertion  is  a  logical  expression  specifying  a  program  state  that  must  exist,  or  a  set 
of  conditions  that  program  variables  must  satisfy,  at  a  particular  point  during  program  ex¬ 
ecution.  Assertion  analysis  is  used  to  determine  whether  program  execution  is  proceeding 
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as  intended.  In  some  cases,  it  may  be  desirable  to  leave  assertions  permanently  in  the  code 
to  provide  a  self-checking  capability.  When  present  in  code,  even  if  commented  out,  asser¬ 
tions  can  provide  valuable  documentation  of  intent. 

Of  the  examined  tools,  only  LDRA  Testbed  currently  supports  dynamic  assertion  anal¬ 
ysis.  Assertions  are  embedded  in  Ada  comments  and  can  be  used  to  (1)  specify  pre-  and 
post-conditions  for  a  section  of  code,  (2)  check  whether  inputs  satisfy  pre-determined  rang¬ 
es,  and  (3)  check  whether  loop  and  array  indices  are  within  bounds.  Should  any  assertion 
fail,  a  user-tailorable  failure  handling  routine  is  executed.  Assertion  checking  can  be 
switched  on  or  off,  allowing  assertions  to  remain  permanendy  in  the  code. 


8.2  Coverage  Analysis 

Coverage  analysis  is  the  process  of  determining  whether  particular  parts  of  a  program 
have  been  exercised.  Its  importance  is  illustrated  by  academic  studies  and  the  experience 
of  the  software  testing  industry  that  have  shown  that  the  average  testing  group  that  does  not 
use  a  coverage  analyzer  exercises  only  50%  of  the  logical  program  structure.  As  much  as 
half  the  code  is  untested  and  therefore  many  errors  may  go  undetected  at  the  time  of  release. 
By  identifying  those  parts  of  a  program  that  have  not  yet  been  executed,  a  coverage  analyz¬ 
er  can  help  to  ensure  that  all  code  is  exercised,  thus  increasing  confidence  in  correct  soft¬ 
ware  operation.  By  measuring  the  coverage  achieved  during  execution  with  particular 
set(s)  of  test  data,  these  tools  also  provide  a  quantitative  measure  of  test  completeness. 
Some  tools  also  aid  in  determining  the  test  data  needed  to  increase  the  coverage.  Although 
coverage  analyzers  do  not  directly  measure  software  correctness,  they  are  valuable  tools  for 
guiding  the  testing  process  and  monitoring  its  progress. 

There  are  two  basic  types  of  coverage  analyzers.  Intrusive  analyzers  instrument  code 
with  special  statements,  called  probes,  that  record  the  execution  of  a  particular  structural 
program  element.  The  addition  of  extra  code  in  the  program  incurs  both  a  size  and  timing 
overhead.  The  alternative,  non-intrusive  analyzers,  requires  special  hardware  and  is  not  ad¬ 
dressed  in  this  report. 


8.2.1  Structural  Coverage  Analysis 

Several  levels  of  structural  test  coverage  have  been  proposed.  The  basic  levels  for  unit 
testing  are  statement,  branch,  and  path  coverage  which  require,  respectively,  each  state- 
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ment,  branch,  or  path  to  be  executed  at  least  once.  They  impose  increasingly  stringent  lev¬ 
els  of  testing  with  statement  coverage  being  the  weakest  and  path  coverage  the  strongest. 
Since  path  coverage  can  be  difficult  to  achieve,  various  additional  levels  that  lie  between 
branch  and  path  coverage  have  been  proposed.  The  best  known  of  these  additional  levels 
are  McCabe’s  Structured  Testing  and  Linear  Code  Sequence  and  Jumps  (LCSAJs)  [Hen- 
nell  1976]. 

Although  unit-level  measures  can  be  applied  during  integration  and  system  testing,  they 
do  not  provide  the  additional  information  that  is  pertinent  at  these  levels.  During  integration 
testing,  for  example,  a  measure  of  the  extent  to  which  the  relationships  between  calling  and 
called  units  has  been  executed  is  useful.  Functional  measures  provide  a  more  appropriate 
measure  of  test  coverage  for  system  testing  (see  Section  8.2.3). 

Table  8-2  summarizes  the  structural  coverage  analysis  features  of  the  examined  tools. 
As  shown  in  this  table,  the  examined  tools  vary  considerably  in  the  support  they  provide. 
The  requirements  for  a  test  driver  to  execute  the  instrumented  program  is  one  of  these  dif¬ 
ferences.  LDRA  Testbed  and  TCMON  automatically  generate  this  test  driver,  as  does 
TestGen  under  certain  circumstances.  The  generated  test  drivers  also  differ.  For  example, 
TCMON  provides  a  command-driven  test  driver  that  allows  the  user  to  explicitly  control 
the  handling  of  generated  trace  files.  Where  necessary,  both  LDRA  Testbed  and  TCMON 
allow  special  actions  so  that  this  interface  can  be  omitted.  There  are  other  significant  dif¬ 
ferences.  For  example,  LDRA  Testbed  provides  different  handling  of  trace  data  to  support 
host/target  testing.  It  also  separates  out  the  data  collected  from  a  concurrent  program  to  al¬ 
low  separate  reporting  for  each  task. 


8.2.2  Data  Flow  Coverage  Analysis 

Data  flow  coverage  has  been  proposed  as  another  measure  of  test  data  adequacy.  While 
the  traditional  structural  coverage  testing  approach  is  based  on  the  concept  that  all  of  the 
code  must  be  executed  to  have  confidence  in  its  correct  operation,  data  flow  testing  is  based 
on  the  concept  that  all  of  the  program  variables  must  be  exercised. 

While  there  are  several  tools  that  provide  this  capability  for  C  programs,  production 
quality  tools  for  data  flow  testing  of  Ada  code  are  not  yet  available.  The  data  flow  testing 
capability  of  LDRA  Testbed,  however,  is  currently  under  beta  testing. 
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Table  8*2.  Structural  Coverage  Analysis  Characteristics 


TOOL  NAME 

Unit-Level  Coverage 

Integration  Coverage  | 

Reporting 

Inst  Files  Differently  j 

Requires  Driver  j 

Limit  Instrumentation  j 

Host/Target  Support  | 

User-control  of  Trace  Files  j 

Statement  Coverage 

Branch  Coverage 

LCSAJ  Coverage 

Condition  Coverage 

McCabe  Coverage 

Path  Coverage 

User-specified  Coverage 

Accumulate  Coverage 

Animation  of  Coverage 

Frequency  Distribution 

Warning  Level 

Identify  (Not)  Hit  Items 

AdaQucst 

El 

El 

El 

El 

El 

z 

El 

El 

LDRA  Testbed 

D 

El 

El 

El 

El 

El 

El 

El 

El 

El 

El 

El 

Logi  scope 

El 

El 

El 

El 

El 

El 

El 

El 

El 

S-TCAT 

El 

El 

El 

El 

El 

El 

El 

TCAT 

El 

El 

El 

El 

El 

El 

El 

El 

TCAT-PATH 

El 

El 

El 

El 

El 

_J 

TCMON 

El 

El 

El 

El 

El 

El 

El 

El 

El 

El 

El 

El 

El 

El 

TestGen 

El 

El 

El 

El 

_ 

El 

_ 

_ 

El 

El 

El 

1 .  Used  in  conjunction  with  TCAT,  TCAT-PATH,  or  S-TCAT  to  animate  coverage  results. 


8.2.3  Functional  Coverage  Analysis 

Functional  coverage,  which  may  also  be  called  requirements  coverage,  provides  a  mea¬ 
sure  of  the  extent  to  which  tests  have  caused  execution  of  the  functions  that  the  software  is 
required  to  perform.  Unlike  structural  tests,  functional  tests  can  determine  problems  such 
as  the  absence  of  needed  code. 

Two  of  the  examined  tools  assess  the  functional  coverage  of  tests.  SoftTest  provides  a 
measure  of  test  adequacy  in  terms  of  the  number  of  tested  functional  variations  with  respect 
to  the  number  of  those  testable.  T  provides  a  measure  of  test  adequacy  based  on  require¬ 
ments  coverage  using  user  specified  pass/fail  results.  An  additional  test  comprehensiveness 
measure  considers  requirements  coverage,  input  domain  coverage,  output  range  coverage 
and,  optionally,  structural  coverage,  where  each  factor  can  be  user-weighted. 
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8.3  Profiling 

Profiling  provides  a  trace  of  the  flow  of  control  during  software  execution.  This  infor¬ 
mation  can  aid  in  locating  the  cause  of  a  failure  and  the  position  of  the  associated  defect. 
Of  the  examined  tools,  both  LDRA  Testbed  and  TST  provides  this  capability  as  an  optional 
feature.  In  the  case  of  LDRA  Testbed,  however,  the  Testbed  may  override  the  user  request 
if  the  resulting  display  exceeds  a  preset  limit 

In  general,  the  majority  of  computing  time  is  incurred  by  only  a  few  program  segments. 
This  may  be  because  these  segments  are  called  frequently,  are  computationally  intensive, 
or  both.  When  a  program  needs  to  be  optimized,  therefore,  it  is  more  efficient  to  start  by 
identifying  where  the  majority  of  computing  time  is  spent  so  that  the  optimization  effort 
can  be  appropriately  focused.  Information  on  the  number  of  times  particular  program  seg¬ 
ments  are  executed  can  aid  this  determination.  The  coverage  analysis  tools  all  give  the 
number  of  times  examined  program  elements  are  executed,  some  additionally  identify  the 
number  of  times  each  program  unit  is  invoked. 

8.4  Timing  Analysis 

Timing  analysis  serves  several  purposes.  These  range  from  supporting  the  validation  of 
software  requirements  that  impose  specific  timing  constraints  on  software  functions  to 
identifying  particular  program  units  that  consume  a  significant  proportion  of  computing 
time. 

AdaQuest  and  TCMON  provide  timing  analysis.  Both  offer  the  flexibility  of  user-spec¬ 
ified  placement  of  timers,  and  measurement  using  either  clock  or  wall  time.  TCMON  ad¬ 
ditionally  allows  a  user  to  request  automatic  timer  instrumentation  at  the  program  unit 
level.  This  tool  reports  on  the  placement  of  timers  (and  any  counters  used  for  structural  cov¬ 
erage  analysis)  to  provide  information  that  can  be  used  to  estimate  the  influence  of  instru¬ 
mentation  statements  on  measured  time. 

8.5  Test  Bed  Generation 

Unit  and  integration  testing  require  the  ability  to  invoke  the  appropriate  modules,  pass¬ 
ing  necessary  inputs  and  capturing  the  actual  outputs  so  that  they  can  be  compared  against 
expected  outputs.  Integration  testing  may  proceed  in  either  a  top-down  or  bottom-up  man- 
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ner.  In  the  first  case,  testing  starts  with  the  most  abstract,  or  high-level  modules  and  requires 
the  use  of  stubs  to  represent  those  modules  called  by  the  module  under  test.  In  bottom-up 
testing,  the  most  detailed,  or  lower-level,  modules  are  tested  first.  Here  test  drivers  are  re¬ 
quired  to  simulate  the  modules  that  invoke  the  modules  under  test.  Development  of  such 
test  drivers  and  stubs  can  be  complex  and  greatly  facilitated  by  automated  support.  In  ad¬ 
dition  to  eliminating  the  need  for  much  manual  labor,  automatic  generation  also  promotes 
a  standardized  testing  environment 

LDRA  Testbed,  TCMON,  and  TestGen  all  generate  the  test  drivers  needed  for  execu¬ 
tion  of  an  instrumented  program.  These  are,  however,  very  limited  drivers  primarily  intend¬ 
ed  to  handle  the  trace  files  used  to  collect  coverage  details.  Of  the  examined  tools,  TBGEN 
and  TST  are  the  only  ones  that  provide  true  test  bed  generation,  and  only  TBGEN  supports 
stub  generation.  Table  8-3  summarizes  the  test  bed  generation  characteristics  c'*  these  two 
tools. 


Table  8-3.  Test  Bed  Generation  Characteristics 


TOOL  NAME 

Driver  Generation  | 

a 

o 

•u 

2 

u 

s 

v 

55 

.e 

s 

V3 

Restrict  Included  Entities  | 

Support  Unknown  Entities  j 

Command 

Language 

Record  Keeping 

Output  Assertions 

Test  Bed  Variables 

Control  Constructs 

Expressions 

Breaks 

Execution  Log 

I/O  Tracing 

Execution  Statistics 

Scripting 

User  Input  Recording 

TBGEN 

T 

3 

TST 

'T 

u 

8.6  Test  Data  Generation  Support 

Dynamic  analysis  requires  software  to  be  executed  with  a  set  of  test  data.  The  resulting 
outputs  are  then  captured  and  compared  with  the  outputs  expected  for  the  given  input  data. 
The  traditionally  manual  and  labor-intensive  method  of  preparing  test  data  has  typically 
limited  the  extent  of  testing  that  is  performed.  Although  the  available  tools  do  not  totally 
replace  the  human  effort  required,  they  can  make  a  substantial  reduction  to  the  amount  of 
human  labor  needed. 
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As  mentioned  above,  dynamic  analysis  requires  comparing  expected  results  against  ac¬ 
tual  results  to  determine  the  success  or  failure  of  a  test.  Determining  expected  results  is  an¬ 
other  traditionally  manual  and  difficult  task.  Research  into  tools,  called  oracles,  to 
automate  this  task  has  been  ongoing  for  many  years.  As  yet,  however,  symbolic  evaluators 
(see  Section  7.9)  come  the  closest  to  supporting  this  capability. 


8.6.1  Structural  Test  Data  Generation 

During  testing,  there  are  occasions  where  it  is  necessary  to  determine  the  test  data  that 
will  cause  a  specific  branch  or  path  to  be  executed.  This  occurs,  for  example,  when  it  is  nec¬ 
essary  to  attain  a  specified  level  of  structural  coverage  and  existing  test  data  has  not  exe¬ 
cuted  some  structural  elements. 

Support  for  this  activity  is  available  at  two  levels.  AdaQuest  and  TCAT  explicitly  iden¬ 
tify  the  program  segments  that  comprise  particular  program  branches  and  paths.  LDRA 
Testbed,  Logiscope,  TCAT-PATH,  and  TestGen  provide  the  same  information  and,  addi¬ 
tionally,  explicidy  identify  the  conditions  required  to  cause  each  structural  element  to  be 
executed. 

8.6.2  Functional  Test  Data  Generation 

Functional  tests  can  be  derived  from  a  requirements  specifications  using  three  catego¬ 
ries  of  methods:  (1)  algorithmic  techniques  such  as  cause-effect  graphing,  equivalence 
class  partitioning,  and  boundary  value  analysis;  (2)  heuristic  techniques  including  fault  di¬ 
rected  testing  and  the  traditional  error  guessing;  and  (3)  random  techniques  that  employ 
random  generation  of  test  data. 

T  supports  all  these  techniques.  Additionally,  it  is  capable  of  incremental  test  data  gen¬ 
eration,  that  is,  tests  can  be  generated  for  software  changes  only.  T  is  the  only  examined 
tool  that  produces  test  data  values  ready  for  immediate  use  in  testing. 

SoftTest  supports  cause-effect  graphing  to  compile  a  database  of  input  conditions  for 
each  unique  function.  The  user  then  works  from  these  conditions  to  determine  the  necessary 
test  data.  In  those  cases  where  identified  functions  are  not  directly  testable,  for  example, 
because  results  produced  by  one  function  may  be  obscured  by  other  functions,  SoftTest 
identifies  intermediate  results  that,  if  observable,  would  enable  otherwise  obscured  func¬ 
tions  to  be  tested. 
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8.6 3  Parameter  Test  Data  Generation 

Thorough  test  coverage  at  the  integration  level  requires  that  each  subprogram  be  exe¬ 
cuted  over  a  range  of  parameter  values.  Of  the  examined  tools,  only  TST  provides  automat¬ 
ed  generation  of  test  data  for  certain  types  of  subprogram  parameters.  This  generation 
occurs  in  one  of  two  forms.  The  user  can  specify  that  all  possible  values  for  a  parameter  be 
generated  (or  first  and  last  values  for  floating  point  numbers).  Alternatively,  the  user  can 
request  that  these  values  are  divided  into  a  number  of  partitions  and  that  the  first,  middle, 
and  last  values  from  each  partition  be  selected. 


8.6.4  Grammar-based  Test  Data  Generation 

In  those  cases  when  the  test  data  is  simply  structured,  and  this  structure  is  amenable  to 
description,  grammar-based  test  data  generation  allows  rapid,  automated  generation  of 
large  amounts  of  test  data.  This  capability  is  particularly  useful  in  random  testing. 

TDGen  provides  this  functionality.  Test  data  is  generated  according  to  location-specific 
data,  uniformly  distributed  data,  or  value-factored  data.  TDGen  can  generate  data  random¬ 
ly,  sequentially,  or  according  to  a  user  specification. 


8.7  Test  Data  Analysis 

Two  types  of  test  data  analysis  are  considered  here.  In  the  first  case,  test  data  sets  are 
analyzed  to  identify  which  test  data  sets  execute  which  lines  of  code.  When  particular  lines 
of  code  are  changed,  this  information  shows  which  test  data  sets  are  affected  by  the  change 
and  must  be  rerun.  The  second  type  of  test  data  analysis  detects  and  reports  on  redundant 
test  data  sets.  This  identifies  test  data  sets  that  are  essentially  equivalent  in  effect  and,  there¬ 
fore,  can  be  eliminated  to  reduce  testing  cost  without  affecting  test  effectiveness. 

LDRA  Testbed  is  the  only  identified  tool  that  supports  these  capabilities.  The  analyses 
are  performed  on  data  collected  during  structural  coverage  analysis. 

8.8  Dynamic  Graph  Generation 

A  visual  representation  of  the  execution  flow  of  a  program  can  aid  in  understanding  that 
program  and  diagnosing  the  cause  of  failures.  LDRA  Testbed  and  Logiscope  provide  this 
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facility  at  both  the  unit  and  integration  levels.  TSCOPE  uses  the  outputs  of  TCAT  or 
TCAT-PATH  to  animate  the  execution  coverage  on  a  directed  graph;  and  the  output  of  S- 
TCAT  can  be  used  to  animate  coverage  on  a  call  tree  representation  of  the  program  under 
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9.  FINDINGS 

This  study  examined  a  number  of  software  testing  tools  to  the  extent  necessary  to  gain 
a  feel  for  their  capabilities.  However,  none  of  the  tools  was  examined  in  great  depth.  Only 
tools  supporting  test  management,  problem  reporting,  and  static  and  dynamic  analysis  of 
Ada  code  were  considered.  Categories  of  tools  such  as  regression  analyzers  and  emulators 
were  ignored.  Additionally,  some  promising  tools  that  may  fill  some  of  the  identified  func¬ 
tional  gaps  are  still  awaiting  examination. 


9.1  Status  of  Available  Tools 

Reviews  of  testing  practices  and  tool  usage  reveal  extremely  poor  exploitation  of  avail¬ 
able  testing  tool  support.  In  the  last  ten  years  software  developers  have  placed  much  focus 
on  software  development  tools  and  there  has  been  an  explosion  in  the  availability  of  C  ASE 
systems  and  other  types  of  development  environments.  Only  in  the  last  few  years,  however, 
has  much  attention  been  paid  to  testing  tools.  These  tools  are  now  starting  to  come  m  mar¬ 
ket  in  increasing  numbers.  Even  so,  available  evidence  suggests  that  they  are  seldom  used. 

As  the  number  of  available  testing  tools  has  increased,  some  trends  are  emerging.  Most 
noticeably,  there  is  an  increased  focus  on  test  management  and  a  movement  towards  cus¬ 
tomer-oriented  measures  of  software  quality.  On  the  technical  side,  there  is  a  movement  to¬ 
wards  graphical  user  interfaces  using  windows.  There  is,  however,  no  evidence  of 
increased  standardization  in  terms  of  testing  functionality.  Even  within  one  category,  no 
single  tool  provides  all  desirable  functionality,  and  different  tools  support  different  groups 
of  functions.  These  functional  differences  require  a  potential  user  to  perform  tool  compar¬ 
isons  with  caution  and  to  select  a  tool  very  carefully. 

The  following  findings  relate  to  the  potential  for  the  use  of  testing  tools  in  the  develop¬ 
ment  and  support  of  SDI  software. 

*  Test  management.  Test  management  tools  offer  critically  needed  support  for  test 
planning  and  test  progress  monitoring.  This  category  of  test  tool  is  perhaps  the  latest 
to  come  to  market  The  capabilities  provided  for  capturing  test  plans,  test  procedures, 
and  test  cases  are  generally  similar.  Capabilities  for  capturing  software  requirements, 
tracing  these  to  particular  tests,  and  supporting  change  impact  analysis,  however,  vary 
significantly.  With  the  exception  of  reliability  analysis  tools,  which  are  becoming 
more  common,  progress  monitoring  is  seldom  available  and  primitive.  Similarly,  only 
one  tool  that  supports  cost  reporting  was  identified  and  the  analysis  performed  is  also 
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primitive.  Nevertheless,  the  ability  of  these  tools  to  manage  a  collection  of  test  infor¬ 
mation  is  very  valuable  and,  even  though  its  analysis  could  be  improved,  the  data 
available  from  this  analysis  is  urgently  needed  to  support  the  management  and  docu¬ 
mentation  of  test  activities. 

•  Problem  reporting.  In  addition  to  their  primary  use  in  tracking  identified  software 
problems  and  managing  problem  resolution,  problem  reporting  tools  offer  support  for 
test  management  They  provide  information  on  the  status  and  quality  of  software 
products;  in  particular,  they  capture  the  data  needed  for  software  reliability  modeling. 
This  data  can  also  provide  valuable  insights  into  the  status  and  quality  of  the  software 
development  processes  themselves,  and  so  support  continuous  process  improvement 

Problem  reporting  tools  fall  into  two  classes.  The  network-based  class  of  tools  are  in¬ 
tended  for  use  on  multiple,  geographically  dispersed  projects.  They  offer  specific 
support  for  customer  submission  of  problem  reports  and  provide  automatic  notifica¬ 
tion  of  changes  in  problem  status.  Those  tools  that  provided  problem  reporting  as  one 
part  of  test  management  capabilities  run  on  a  stand-alone  personal  computer  but  cap¬ 
ture  much  of  the  same  types  of  problem  information  and  provide  similar  analyses  of 
problem  data.  There  are  several  problem  reporting  tools  that  could  be  brought  into  im¬ 
mediate  use,  although  some  thought  should  be  given  to  defining  a  standardized  set  of 
problem  data  to  be  captured  across  all  SDI  software  development  efforts. 

♦  Static  analysis.  Available  static  analysis  tools  are  essentially  limited  to  facilitating 
program  understanding  and  assessing  characteristics  of  software  quality.  They  pro¬ 
vide  some  minimal  support  for  guiding  dynamic  testing.  Static  analysis  requires  little 
in  the  way  of  test  environment  set  up  and  a  minimum  of  human  intervention.  It  can 
detect  the  presence  or  absence  of  certain,  limited  types  of  defects  and  allows  these 
defects  to  be  detected  reliably  and  early  in  the  testing  process.  The  types  of  defects 
traditionally  found  by  static  analysis  tools,  however,  are  now  routinely  checked  for 
by  Ada  compilers.  Currently,  one  of  the  main  values  of  static  analysis  tools  is  in  sup¬ 
porting  an  understanding  of  software  and  guiding  dynamic  testing.  Quality  analysis 
is  a  particular  type  of  static  analysis  where  assessment  of  a  set  of  predefined  quality 
characteristics  can  be  used  to  provide  early  indication  of  general  software  quality  and 
the  identification  of  potential  problem  areas.3 

In  the  types  of  tools  examined,  complexity  analysis  and  control  flow  analysis  are  the 
most  common  static  analysis  functions  supported.  A  couple  of  examples  of  data  flow 
analysis  tools  have  appeared  and  are  expected  to  become  more  common  in  the  future. 
Two  types  of  tools  to  aid  a  user  in  understanding  and  documenting  a  piece  of  code  are 
available:  graph  generators  and  browsers.  Flow  graph  and  call  graph  generations  are 

3.  The  role  of  quality  analysis  is  discussed  extensively  in  the  GPALS  Software  Quality  Program  Plan 

[GPALS  1992a]. 
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quite  common,  although  they  vary  greatly  in  the  quality  of  the  representations  used  to 
present  these  graphs.  A  few  browsers  are  currently  available  and  these  are  expected 
to  become  more  common  over  the  next  few  years.  This  study  and  the  CRWG’s  study 
of  quality  analysis  tools  have,  between  them,  identified  several  tools  that  check  con¬ 
formance  of  code  with  a  set  of  project  standards.  One  of  these,  ADAMAT,  checks  for 
conformance  with  the  SPC  Ada  style  guidelines.  Two  new  tools  that  also  support  the 
SPC  standards  have  recently  been  identified.  More  advanced  types  of  static  analysis, 
such  as  symbolic  evaluation,  are  uncommon. 

•  Dynamic  analysis.  Although  many  needed  dynamic  analysis  capabilities  are  infre¬ 
quently  available,  tools  are  available  that  offer  considerable  support  for  dynamic  test¬ 
ing  to  increase  confidence  in  correct  software  operation.  Dynamic  analysis  is  the 
principle  method  used  for  software  validation  and  verification.  Here  automated  sup¬ 
port  for  the  preparation  of  a  test  bed,  generation  of  test  data,  and  analysis  of  test 
results  is  needed.  Tools  that  provide  this  functionality  will  decrease  the  cost  of  testing 
by  increasing  the  productivity  of  the  human  tester  and  increase  software  quality  by 
supporting  such  activities  as  test  data  adequacy  assessment 

Structural  coverage  analyzers  and  profilers  are  the  most  common  dynamic  analysis 
tools  and  are  widely  available  on  a  range  of  operating  platforms.  The  structural  cov¬ 
erage  analyzers  generally  focus  on  statement  and  branch  coverage,  that  is,  relatively 
low  coverage  measures.  Support  for  path  coverage  analysis  and  structural  coverage 
at  the  integration  level  is  less  frequently  available. 

Support  for  other  types  of  dynamic  analysis  is  also  infrequently  available.  Only  two 
of  the  examined  tools  provide  timing  analysis.  Only  two  tools  offer  test  driver  gener¬ 
ation  for  bottom-up  testing,  and  only  one  of  these  also  generates  the  stubs  needed  for 
top-down  testing.  Few  tools  support  test  data  generation  for  structural  or  random  test¬ 
ing,  although  two  tools  that  support  the  generation  of  functional  test  data  from  a  re¬ 
quirements  specification  have  been  introduced.  Assertion  testing  is  a  relatively  new 
capability  that  is,  as  yet,  only  provided  by  one  tool. 

Tools  of  similar  types  vary  widely  in  the  capabilities  they  provide  and  in  characteristics 
such  as  tailorability  and  robustness.  In  general,  the  examined  tools  require  little  sophistica¬ 
tion  on  the  part  of  the  user  and  are  supported  by  good  documentation.  Some  actively  guide 
a  user  through  necessary  tasks,  keep  a  record  of  test  activities,  and  take  extra  steps  to  relieve 
the  user  of  repetitive  tasks.  In  general,  however,  the  tools  employ  primitive  user  interfaces 
that  could  benefit  from  the  application  of  human  factors  engineering.  In  several  cases,  the 
need  to  refer  to  separate  listings  to  identify  objects  referenced  in  reports  complicated  tool 
use.  There  were  instances  where  different  tools  gave  different  results  when  performing  the 
same  function,  for  example,  calculating  cyclomatic  complexity.  Moreover,  some  of  the 
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tools  contained  faults.  While  most  failures  were  trivial,  others  rendered  a  tool  unusable  un¬ 
til  fixed  by  the  supplier.  In  three  cases,  major  failures  occurred  when  using  the  tool  on  sam¬ 
ple  software  supplied  by  the  supplier.  Consequently,  prospective  tool  users  should  carefully 
consider  a  tool’s  usage  history  and  the  types  of  support  options  provided  by  the  tool  sup¬ 
plier. 


9.2  Significant  Deficiencies 

Available  testing  tools  offer  significant  opportunities  for  increasing  software  quality 
and  reducing  development  and  support  costs.  Even  so,  there  are  a  number  of  problems  with 
these  tools  and  a  lack  of  needed  functionality  that  may  handicap  SDI  software  testing.  The 
following  problems  are  of  particular  concern. 

•  There  is  a  lack  of  support  for  testing  concurrent  Ada  software.  The  vast  majority 
of  current  testing  techniques  are  intended  for  testing  sequential  code.  Concurrent  soft¬ 
ware,  however,  introduces  special  concerns.  The  inherent  indeterminism  of  concur¬ 
rent  programs  means  that  two  executions  of  the  same  software  with  the  same  inputs 
can  produce  different  behaviors.  This  lack  of  reproducibility  handicaps,  for  example, 
determining  the  cause  of  a  failure  and  retesting  a  modified  program.  Concurrent  pro¬ 
grams  can  also  contain  a  new  class  of  faults,  called  synchronization  faults.  Additional 
tests  are  needed  to  check  for  existence  of  these  faults.  Testing  techniques  addressing 
these  issues  are  appearing,  along  with  some  prototype  tools,  such  as  AdaTDC  being 
sponsored  by  the  National  Science  Foundation.  One  commercial  tool  that  is  expected 
to  support  concurrent  re-execution  within  the  next  year  is  PARTAMOS,  under  devel¬ 
opment  by  Alcatel  Austria.  Meanwhile,  the  majority  of  the  commercial  Ada-based 
static  and  dynamic  analyzers  are  capable  of  recognizing  all  the  concurrent  Ada  lan¬ 
guage  features,  but  not  fully  acting  on  them. 

•  There  is  a  need  for  increased  tool  integration  to  provide  more  complete  coverage 
of  testing  activities.  The  majority  of  tools  provide  support  for  a  specific,  limited  set 
of  testing  activities.  No  single  tool,  or  supplier  toolset,  provides  all  desirable  function¬ 
ality.  While  tools  that  support  different  types  of  activities  can  generally  be  used 
together,  simply  applying  them  independently  in  sequence  is  usually  not  the  most 
cost-effective  approach.  It  can  incur  unnecessary  duplication  of  both  human  and  com¬ 
puter  work  and  may  require  additional  steps  to  make  the  output  of  one  tool  acceptable 
to  another.  It  also  requires  users  to  gain  familiarity  with  a  number  of  different  user 
interfaces,  as  well  as  requiring  system  administrators  to  support  a  number  of  indepen¬ 
dent  tools.  Moreover,  true  functional  integration  requires  some  common,  underlying 
model  of  the  software  development  process  model.  For  example,  a  test  log  automati- 
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cally  captured  by  a  test  bed  during  test  execution  activities  should  be  the  same  log  that 
a  test  management  tools  uses  in  monitoring  test  progress.  This  type  of  integration 
would  greatly  increase  the  power  of  available  tools  and  their  ease  of  use. 

•  There  is  a  need  for  integration  of  testing  tools  into  CASE  systems  to  provide 
improved  feedback  into  development  activities.  While  some  CASE  systems  do 
provide  support  for  code  level  testing,  this  support  is  generally  less  extensive  than  that 
provided  by  stand-alone  testing  tools.  At  the  same  time,  CASE  tools  are  providing 
more  support  for  testing  activities  during  early  development  phases  than  stand-alone 
tools.  A  more  careful  look  at  the  testing  capabilities  of  current  CASE  systems  is 
needed,  together  with  an  evaluation  as  to  which,  and  how,  independent  tools  should 
be  integrated  with  them  to  provide  a  comprehensive  test  environment.  Here  again, 
functional  integration  into  a  CASE  requires  that  test  processes  and  products  are  them¬ 
selves  integrated  into  the  underlying  software  development  process  model.  These 
issues  have  yet  to  be  addressed. 

There  is  a  lack  of  data  on  the  cost  effectiveness  of  particular  test  techniques  and  tools 
that  can  be  used  to  encourage  and  guide  their  use.  Although  there  have  been  many  studies 
into  the  comparative  value  of  certain  test  techniques,  there  is  a  lack  of  data  on  the  practical 
costs  and  benefits  of  particular  testing  techniques,  and  the  tools  that  support  those  tech¬ 
niques.  This  information  is  needed  to  determine,  for  a  given  set  of  circumstances,  the  most 
appropriate  techniques  and  tools  to  apply,  the  order  in  which  to  apply  them,  and  the  extent 
of  that  use.  It  is  also  needed  during  planning  activities,  to  support  the  estimation  of  needed 
testing  resources,  and  in  monitoring  test  progress.  The  data  captured  in  test  logs  and  prob¬ 
lem  reports  can  be  used  for  this  purpose,  imposing  a  minimal  data  collection  burden  on  soft¬ 
ware  developers.  Where  this  data  is  maintained  automatically,  it  will  be  a  simple  task  to 
forward  it  to  a  central  site  for  analysis,  such  as  the  Level  2  System  Simulator  (L2SS)  soft¬ 
ware  metrics  database. 

A  number  of  promising  testing  techniques  have  been  proposed  in  the  last  decade  that 
have  failed  to  progress  beyond  prototype  status.  One  example  of  this  is  the  software  fault 
tree  analysis  used  for  error  cause  and  effect  analysis  in  support  of  risk  management  The 
Anna  toolset  is  an  example  of  a  suite  of  prototype  tools  for  assertion-based  testing  of  Ada 
code.  Further  development  of  such  techniques  and  supporting  tools  could  start  to  fill  some 
of  the  gaps  in  needed  testing  capabilities. 

Additionally,  there  are  needed  automated  test  capabilities  that  are  provided  for  other 
languages  but  not  available  for  Ada.  Examples  of  capabilities  available  for  other  languages 
include  error  seeding  as  another  measure  of  test  data  adequacy;  support  for  test  coverage 
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analysis  of  kernel,  daemon,  and  library  code  in  addition  to  application  code;  and  critical 
path  analysis.  Similarly,  while  several  tools  supporting  data  flow  testing  of  C  code  are 
available,  only  one  tool  supplier  with  plans  to  provide  data  flow  testing  of  Ada  code  has 
been  identified.  Here  again,  further  tool  development  could  start  to  fill  some  of  the  gaps  in 
needed  testing  capabilities. 
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10.  INTRODUCTION 

This  part  of  the  report  describes  the  selected  tools  in  terms  of  their  usage.  Tools  are 
grouped  by  supplier  and  the  report  details  the  operating  environment  and  the  functionality 
provided.  Where  applicable,  price  information,  accurate  at  the  time  of  examination,  is  also 
included.  Each  description  is  supported  with  observations  on  ease  of  use,  documentation 
and  user  support,  and  Ada  restrictions.  Problems  encountered  during  the  examinations  pro¬ 
vided  insight  into  the  reliability  and  robustness  of  each  tool.  Each  description  is  accompa¬ 
nied  by  sample  outputs. 

Table  10-1  summarizes  the  details  given  for  each  tool.  It  also  identifies  available  bridg¬ 
es  between  testing  tools  and  CASE  systems.  Table  10-2  presents  relevant  supplier  data. 
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-  Users  C  -  Command  driven  *  -  Price  for  3-day  training  course 

-  Sites  B  -  Menu  and  command  driven  b  -  Sold  as  group 

-  Optional 


Table  10*2.  Supplier  Profiles 


Introduction 


PART  II 


RELATED  SUPPLIER  TOOLS 

3SV3 

™ 1 

■ 

V 

■ 

09 

siSi<|Bav  aoiss&rifeu 

v  U- 

LU 

a 

sisX|Buy  jtmta^a 

?•  > 

~r 

UU  uu 

i 

V  V 

sis/Cpay  apeis 

v  > 

■ 

V 

V 

V  V 

gapjotbu  oaaiqojj 

v 

■ 

u. 

lu9U»8eaew  jsaj. 

LU 

■ 

V 

1 

Tool  Name 

1 

Runner 

QES/Architect 

QES/Qease 

QES/Expert 

QES/Programmer 

RDDTs 

QTET 

SQA:  Robot 

SMARTS 

CAPBAK 

EXDIFF 

STATIC 

TRACKER 

l 

o 

£ 

Ui 

oo 

< 

AGE/ASA 

AGE/GEODE 

DocBuilder 

Examined  Tools 

< 

a 

LU 

AutoFlow-Ada 

SoftTest 

fc* 

V 

00 

c0 

*  § 

o  ^ 
>»  v* 

£  z 

AdaQuest 

LDRA  Testbed 

QES/Manager 

DDTs 

MALPAS 

H 

Ul 

t- 

T-PLAN 

SQA:  Manager 

jSRE  Toolkit 

TCAT,  S-TCAT, 
TCAT-PATH, 
TDGen,  TSCOPE 

TestGen,  QualGen, 
GrafBrowse, 
ADADL  Processor 

[TBGEN,  TCMON 

Logiscope 

SERVICES 

PROVIDED 

jjoddns  ami-joH 

a 

a 

a 

V 

a 

a 

V 

a 

1 

a 

■ 

v 

>• 

V 

jat)a|SM3M 

a 

■ 

V 

a 

a 

■ 

a 

3 

■ 

v 

dnu<)  jasf| 

■ 

■ 

V 

a 

■ 

V 

a 

9 

■ 

>■ 

u. 

a 

■ 

m 

a 

■ 

■ 

a 

V 

AmraiMlNl 

a 

a 

m 

■ 

■ 

T 

a 

a 

a 

MESS 

a 

a 

V 

m 

a 

a 

T 

~r 

a 

a 

a 

v 

V 

V 

jfDOBJinsno^ 

a 

a 

a 

“> 

- 

a 

■ 

V 

V 

a 

a 

a 

a 

> 

V 

SUPPUER  STATISTICS 

paqsiiqeisa 

00 

Os 

00 

oo 

o 

• 

r- 

r- 

Os 

00 

£ 

Os 

1960s 

r** 

00 

2 

00 

o 

Os 

Os 

o 

00 

OS 

Os 

Os 

00 

O' 

s 

2 

Os 

00 

OS 

tr> 

00 

Os 

P* 

r- 

2 

»r> 

00 

2 

<N 

Os 

2 

VO 

00 

Os 

anS  jaqddns 

V 

o 

V 

o 

V 

>3,000 

-1,000 

o 

N 

o 

<N 

O' 

00 

in 

»n 

m 

M 

A 

o 
*— < 

V 

tn 

V 

— 1 

o 

<N 

V 

Phone  Number 

! 

3 

ro 

r- 

<N 

2 

5 

oo 

O 

X 

vO 

O' 

S 

O' 

^r 

(708)  574-3030 

3 

r* 

s 

Os 

«r> 

o 

00 

00 

<N 

00 

00 

<N 

*n 

sA 

8 

I 

(908)918-0110 

l 

(408)  274-8867 

5 

r- 

• 

<s 

1 

r- 

00 

O' 

* 

«n 

s 

tr\ 

V) 

5 

e'¬ 

en 

r- 

<N 

O 

a 

OS 

O' 

• 

00 

n 

C4 

8 

00 

00 

e'¬ 

en 

00 

n 

§ 

oo 

(415)957-1441 

3 

*A 

s© 

? 

£ 

sO 

Tf 

sn 

2 

m 

00 

\Tt 

m 

+ 

(214)  241-6595 

SUPPLIER  NAME 

1 

lAutoCASE  Technology 

■Bender  &  Associates 

■Computer  Power  Group,  Inc. 

iGeneral  Research  Corp. 

■Program  Analysers,  Ltd. 

I Programming  Environments,  Inc. 

Quality  Engineering  Software, 

Inc. 

■QualTrak  Corp. 

ITA  Consultancy  Services,  Ltd. 

ISTARS  Foundation  Repository 

■Software  Quality  Assurance,  Ltd. 

ISoftware  Quality  Automation 

■Software  Quality  Engineering 

Software  Research,  Inc. 

Software  Systems  Design 

■Testwell  Oy 

Verilog  USA 

U- 


10-4 


-  Future  capability 


PART  II 


AdaQuest 


11.  AdaQuest 

The  AdaQuest  toolset  provides  a  variety  of  static  and  dynamic  techniques  for  testing 
Ada  software.  It  is  based  on  two  earlier  verification  and  validation  systems,  RXVP80  and 
J7AVS,  that,  respectively,  support  Fortran  and  Jovial  testing. 

The  static  analysis  capabilities  of  the  current  version  of  AdaQuest  are  limited  to  iden¬ 
tifying  program  branches  and  the  lexical  nesting  structure  of  specified  compilation  units. 
Existing  dynamic  capabilities  consist  of  coverage  and  timing  analysis. 


11.1  Tool  Overview 

AdaQuest  was  developed  by  General  Research  Corporation.  The  first  version  of  this 
toolset,  version  1.1,  became  available  in  December  1991.  It  runs  on  VAX/VMS  platforms. 
At  the  time  of  evaluation,  the  price  for  AdaQuest  started  at  $6,500. 

AdaQuest  requires  that  code  to  be  analyzed  resides  in  an  AdaQuest  program  library. 
Each  library  is  associated  with  a  VMS  directory  that  contains  the  intermediate  files  of  the 
relevant  compilation  units.  Several  library  management  commands  are  provided.  These  in¬ 
clude  commands  to  set  a  current  library  and  build  a  working  set  of  compilation  units.  Spe¬ 
cial  facilities  are  provided  for  reading  source  files  into  a  library;  in  the  current  version  of 
the  toolset,  source  files  arc  limited  to  containing  a  single  compilation  unit. 

The  AdaQuest  Analyzer  generates  branch  reports  and  unit  nesting  reports  for  user-spec¬ 
ified  library  units.  In  the  first  case,  the  result  is  an  annotated  source  code  listing  that  iden¬ 
tifies  and  numbers  each  decision  branch  in  each  program  unit  of  a  specified  compilation 
unit.  This  report  is  needed  to  select  locations  for  the  insertion  of  coverage  and  timing  probes 
(see  below).  It  is  also  required  for  interpretation  of  branch  coverage  reports.  The  unit  nest¬ 
ing  report  shows  the  lexical  nesting  of  the  program  units  in  a  compilation  unit. 

Each  program  unit  can  be  instrumented  to  collect  either  coverage  data,  timing  data,  or 
both.  The  user  specifies  the  library  unit  bodies  and  subunits  to  be  instrumented,  and  each 
instrumented  unit  is  written  to  a  separate  file.  Instrumentation  is  performed  by  inserting 
special  statements  into  the  source  code.  Where  necessary,  individual  source  code  state¬ 
ments  are  first  transformed  so  that  these  insertions  will  be  syntactically  legal.  An  exit  state¬ 
ment,  for  example,  may  be  replaced  with  if  and  goto  statements. 
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Two  different  types  of  probes  are  available  to  collect  coverage  data.  Branch  coverage 
probes  are  inserted  automatically  at  each  branch  point  (including  at  the  start  of  each  accept 
and  block  statement).  Any  code  transformations  necessary  to  ensure  correct  coverage  mea¬ 
surement  are  also  made  automatically.  The  second  type  of  probe,  called  test  case  probes, 
are  used  to  partition  the  data  collected  from  an  instrumented  program.  They  allow,  for  ex¬ 
ample,  measuring  the  coverage  achieved  in  each  execution  of  a  loop.  Test  case  probes  are 
inserted  at  user-specified  points  in  the  source  code  and  take  the  form  of  procedure  call  state¬ 
ments.  It  is  the  user’s  responsibility  to  ensure  that  these  are  placed  in  a  syntactically  legal 
fashion;  AdaQuest  does  not  check  the  placement.  The  resulting  instrumented  files  include 
file  header  information  that  identifies  the  unit,  original  source  file,  and  type  of  instrumen¬ 
tation  performed.  They  are  accompanied  by  files  containing  two  additional  AdaQuest-gen- 
erated  units  needed  for  the  collection  of  coverage  data. 

For  timing  analysis,  probes  are  also  inserted  at  user-specified  locations.  In  each  case, 
the  user  gives  a  start  and  stop  location  in  the  form  of  source  code  line  numbers;  these  loca¬ 
tions  may  reside  in  different  program  units  within  a  compilation  unit  Again,  it  is  the  user’s 
responsibility  to  ensure  that  insertions  at  these  defined  locations  will  be  syntactically  legal. 
The  user  also  specifies  whether  data  should  be  measured  in  terms  of  CPU  or  wall  clock 
time. 

Compilation  and  linking  of  the  instrumented  program  is  performed  using  the  standard 
VAX  facilities.  AdaQuest  does,  however,  provide  a  compilation  script  that  can  be  used  to 
compile  the  two  AdaQuest-generated  run-time  units.  When  executed,  the  instrumented  pro¬ 
gram  collects  coverage  and  timing  data  in  an  automatically  created  trace  file.  (As  with  the 
other  tools  that  write  coverage  details  to  a  trace  file,  a  program  run  must  terminate  normally 
so  that  the  trace  file  is  closed  by  the  operating  system.)  If  desired,  the  user  can  allocate  a 
name  and  description  to  the  trace  file. 

AdaQuest  maintains  a  test  history  for  each  body  or  subunit  in  a  library  in  order  to  allow 
reporting  on  the  cumulative  coverage  achieved  over  a  series  of  test  runs.  Initially  empty, 
the  user  specifies  when  a  trace  file  should  be  appended  to  the  appropriate  histories.  Nor¬ 
mally,  the  test  history  for  a  unit  is  cleared  when  the  program  library  is  updated.  In  certain 
circumstances,  the  user  can  override  this  function  to  keep  a  history,  although  this  ability 
must  be  used  with  great  care. 

The  reports  that  are  available  can  be  produced  for  all  or  only  user-specified  program 
units.  The  Test  Run  Report  identifies  the  original  source  code  file(s)  and  indicates  how  it 
was  instrumented.  Coverage  reports  are  generated  using  data  from  a  single  trace  file,  called 
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the  current  test  run,  and,  in  most  cases,  user-specified  test  histories.  Between  them,  the  cov¬ 
erage  reports  provide  counts  of  the  number  of  times  each  program  unit,  accept  statement, 
and  block  statement  was  executed,  counts  of  the  number  of  times  each  conditional  branch 
was  executed,  the  execution  status  (first-time  hit,  never  hit)  of  each  branch,  and  the  percent¬ 
age  coverage  of  the  branches  in  each  unit.  In  some  cases,  histograms  are  provided  to  com¬ 
pare  the  execution  counts  of  different  items.  Two  additional  reports  can  be  generated  using 
coverage  information  from  the  test  history  files  alone.  Timing  analysis  reports  are  generat¬ 
ed  from  a  single  trace  file.  They  detail  the  timing  probe  placement,  the  number  of  times 
each  selected  code  segment  was  invoked,  and  the  minimum,  maximum,  and  average  time 
taken  for  each  segment.  Timing  data  is  not  accumulated  in  test  histories. 


11.2  Observations 

Ease  of  use.  The  user  interacts  with  AdaQuest  through  a  command  interface.  This  in¬ 
terface  requires  considerable  memorization  on  the  user’s  part  (the  Analyzer,  for  example, 
has  some  27  different  commands)  and  exhibits  some  inconsistencies.  Although  the  ability 
to  explicitly  specify  the  locations  for  coverage  and  timing  probes  can  be  valuable,  the  need 
to  manually  refer  to  the  annotated  source  code  listing  is  tedious  and  a  possible  source  of 
error.  This  inconvenience  could  be  reduced  by  providing,  for  example,  some  automatic  in¬ 
sertion  of  timing  probes  to  measure  the  time  spent  in  named  units.  Output  listings  are  han¬ 
dled  in  an  unusual  manner;  all  commands  that  produce  an  output  listing  automatically 
invoke  the  VAX  edlin  editor. 

At  the  time  of  evaluation,  the  on-line  help  provided  summary  information  on  only  a 
small  number  of  available  commands,  although  this  should  be  a  useful  feature  when  com¬ 
pleted. 

Documentation  and  user  support.  A  complete  AdaQuest  user  manual  was  not  avail¬ 
able  at  the  time  of  examination.  The  documentation  that  was  provided,  however,  was  well- 
written  and  easy  to  follow.  One  nice  feature  is  a  command  dictionary  that  provides  a  useful 
reference  manual.  Tool  installation  was  straightforward. 

Instrumentation  overhead.  AdaQuest  allows  the  user  to  control  the  extent  of  instru¬ 
mentation  by  requiring  the  user  to  explicitly  identify  the  units  to  be  instrumented.  Two  spe¬ 
cial  run-time  units  are  provided  that  must  be  included  in  an  instrumented  executable  to 
handle  the  creation  and  recording  of  a  trace  file.  Including  these  special  units,  full  instru- 
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mentation  for  branch  coverage  of  the  Ada  Lexical  Analyzer  Generator  gave  an  approxi¬ 
mately  19%  increase  in  the  total  source  code  size. 

Ada  restrictions.  AdaQuest  supports  the  full  Ada  language,  including  extensions  de¬ 
scribed  in  Chapter  13  of  the  Ada  Language  Reference  Manual.  The  only  exception  is  the 
Ada  terminate  alternative  (see  LRM  9.7.1)  which  contains  no  statements  and  cannot  be  in¬ 
strumented. 

Problems  encountered.  No  problems  were  encountered  during  the  examinations  of 
this  tool.  AdaQuest  operated  exactly  as  described  in  the  documentation  provided. 


11.3  Planned  Additions 

Future  versions  of  the  static  analyzer  are  expected  to  generate  dependency  reports  and 
check  for  logic  errors  (such  as  infinite  loops,  unreachable  statements,  and  uninitialized 
variables).  Conformance  checking  against  standards  relating  to  the  use  of  forbidden  con¬ 
structs  and  those  specifying  maximum/minimum  constraints  on  the  quantity  of  Ada  con¬ 
structs  appearing  within  a  certain  scope  is  also  anticipated.  A  source  code  profiler  will  list 
any  non-zero  counts  for  some  228  Ada  features.  AdaQuest  is  also  expected  to  include  a 
query  facility  that  provides  direct  access  to  this  data  for  quality  analysis  tools. 

Additional  dynamic  analysis  capabilities  expected  to  become  available  include  the  use 
of  assertions  for  checking  unit-  and  interface-level  design  constraints.  Finally,  a  task  ana¬ 
lyzer  is  also  planned  that  traces  the  actual  synchronization  relationships  between  Ada  tasks, 
creating  timing  diagrams  to  help  in  diagnosing  synchronization  errors  such  as  deadlock  and 
starvation. 


11.4  Sample  Outputs 

Figures  11-1  through  11-12  provide  sample  outputs  from  AdaQuest 
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> 


27-JAN-1992  10:45 
Library:  adalex.work 

Structure  Unit 

ADAQUEST  UNIT  NESTING  REPORT 

Coop 

Unit  Kind 

PAGE  1 

Unit:  LL_CQMPILE : BODY 
27- JAN-1992  10:35:12 
Starting  Line 

LL_COMPILE 

Procedure  Body 

26 

. .LLNEXTTOKEN 

Procedure  Spec 

136 

. . LLFIND 

Function  Body 

140 

.  .  LLPRTSTRING 

Procedure  Body 

165 

. . LLPRTTOKEN 

Procedure  Body 

177 

. .LLSKIPTOKEN 

Procedure  Body 

190 

. . LLSKIPNODE 

Procedure  Body 

203 

. . LLSKIPBOTH 

Procedure  Body 

217 

.  .  LLF ATAL 

Procedure  Body 

234 

.  .  GET_CHARACTER 

Procedure  Body 

246 

. .  MAKE  JTOKEN 

Function  Body 

264 

_ CVTSTRING 

Function  Body 

271 

. . LL_TOKENS 

Package  Spec 

326 

. . . .ADVANCE 

Procedure  Spec 

328 

. . LL_T0KENS 

Package  Stub 

334 

. .LLNEXTTOKEN 

Procedure  Body 

337 

.  .  LLTAKEACTION 

Procedure  Stub 

349 

. . LLMAXN 

Procedure  Body 

352 

. . . .KEADGRAM 

Procedure  Body 

383 

. BUILDRIGHT 

Procedure  Body 

389 

Procedure  Body 

449 

. . . .PAUSE 

Procedure  Body 

499 

•••••» ERASE 

Procedure  Body 

505 

. TESTSYNCH 

Procedure  Spec 

522 

. EXPAND 

Procedure  Body 

525 

Function  Body 

533 

. TESTSYNCH 

Procedure  Body 

593 

Procedure  Body 

596 

27-JAN-1992  10*45 

ADAQUEST  UNIT  NESTING  REPORT 

PAGE  2 

Library*  ADALEX/WOKK 

Coap  unit* 

LL_COMPILE .  LLJTOKENS 
27-JAN-1992  10:39:35 

Structure  Unit 

Unit  Kind 

Starting  Line 

LL_TOXENS 

Package  Body 

25 

.  .ADVANCE 

Procedure  Body 

49 

.... GET_CHAR 

Procedure  Body 

56 

_ CHAK_ADVANCE 

Procedure  Body 

69 

- LOO K_ AHEAD 

Procedure  Body 

86 

Figure  I'M.  AdaQuest  Unit  Nesting  Report 
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27-JAN-1992  10:45  ADAQUEST  BRANCH  REPORT  PAGE  1 

Library:  ADALEX ; WORK  Coap  Unit:  LL_COMPILE : BODY 

27-JAN-1992  10.35:12 


140  function  LLPIND(  ITS4:  LLSTRINGS ;  WHICH:  LLSTTLE  )  raturn  INTEGER  la 

141  —  rind  itam  in  aymbol  table  —  return  index  or  0  if  not  found. 

142  —  Aasumea  aymbol  table  ia  aorted  in  aaoending  order. 

143  LOW,  MIDPOINT,  HIGH:  INTEGER; 

144  begin 

'***  BRANCH  1  PROGRAM  UNIT  START 

145  LOW  1; 

146  HIGH  :«  LLTABLESIZE  +  1; 

147  while  LOW  /-  HIGH  loop 

'•••  BRANCH  2  LOOP  TEST  PAIL 
'•**  BRANCH  3  LOOP  TEST  PASS 

148  MIDPOINT  (HIGH  +  LOW)  /  2; 

149  if  ITB1  <  LLSTMBOLTABLE( MIDPOINT) -RET  than 
'**•  BRANCH  4  IF  TEST  PASS 

150  HIGH  MIDPOINT; 

151  elalf  ITEM  -  LLSTMBOLTABLE( MIDPOINT)  .KEY  then 
BRANCH  S  ELSIE  TEST  PASS 

152  if  LL8YMB0LTABLE{ MIDPOINT) .KIND  -  WHICH  then 
<•••  BRANCH  6  IF  TEST  PASS 

726  begin  —  LLCOMPILE 

'•••  BRANCH  142  PROGRAM  UNIT  START 

727  CREATE  <STANDARD_ERROR,  0UT_FILE,  *atd_error*,  *•); 

728  LLMAIN; 

730  CLOSE  (STANDARD_ERROR); 

731  end  LL_COMPILE; 


BRANCH  SUMMARY 

Branch  Kind  Branch  ID  Begin  End  Branch  Statement  Path 


PROGRAM  UNIT  START 

1 

•  145 

147 

145 

146 

147 

LOOP  TEST  PAIL 

2 

147 

161 

147 

161 

LOOP  TEST  PASS 

3 

147 

149 

147 

148 

149 

IP  TEST  PASS 

4 

149 

147 

149 

150 

147 

ELSIF  TEST  PASS 

5 

149 

152 

149 

152 

IP  TEST  PASS 

« 

152 

153 

152 

153 

PROGRAM  UNIT  START 

142 

727 

730 

727 

728 

730 

• 

“  Branch 

containa 

an  Inflnlta  Loop 

Unreachable  Statement a  ->  —  NONE  — 


Figure  1 1-2-  AdaQuest  Branch  Report 
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27- JAN— 1992  12:13 

ADAQUEST  TEST  RUN  REPORT 

PAGE  1 

Traoe  File  : 

Time  of  Run 

Name 

Description 
#  of  Test  Cases  : 

USR : (ADATEST ] ADAQUEST . ETF ; 16 

27-JAN-1992  12:08:39 

EXAMPLE_1 

Run  with  1st  test  file 

1 

Test  Sun  Units  : 

LL_COMPILE : BODY 

Instrumental]  From  File  i 

USRi  (ADATEST  .  AD  ALEX  2  ]  LL_COMPILE .  ADA;  10 
Instrumentation  Parameters  : 

Coverage 

LL_SUPPORT : BODY 

Instrumented  From  File  i 

USR: [ADATEST . AD ALEX 2] LL_SUP_BODT . ADA ; 1 
Instrumentation  Parameters  t 
Coverage 


Figure  11-3.  AdaQuest  Coverage  Test  Run  Report 
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Figure  11*4.  AdaQuest  Unit  Coverage  Report 
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Figure  11*6.  AdaQuest  Branch  Coverage  Summary  Report 
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Figure  11-8.  AdaQuest  Branch  Coverage  Not-HIt  Report 


PART  II 


AdaQuest 


!s  8i 


s3s 


s§ 

a.  O 


£|*  -- 
o;« 


Jri, 
J  I' 
J  X 


:s 

3 


m  «h  m 
x  >j  - 

’*■  .£  ! 

i*ji  : 

J  %  i 

«**! 

9  I  I 

O - « 

•m  ft  • 

o  c  • 
o  ■ 

U  ««  I 

|  s  S 

*S  | 


X  <4 
M  +1 

-JS 

!i 


to  i3 

il 


i  1 
53 


IKS 


?t 

12 
•X  « 
*• 

S3 

72 

a* 


5 

35 

«to  ft. 

o 


J,  I 

C  St 


P.r«nHHmi>>Hin<4iA 


o*ooo«o<«o»r> 


HftnfiHtnnf*! 


i  ill 

SSilSai 

Lillis: 


wlu 


to 

!| 


?! 

73 

s? 

AS 

N  3 


Si 

iS 


+  **  +  ***■+***  ■  M  I 


o  o  o  o  o 


r*  o 
w  o 


^  ^  *4  I 


h  anwrinn  run  Hm 


OHOOOOODOOO 


rimonr 


lOOOOrOf'O'#* 

(  o  e  ft  o  «  o  v  <4 


0  0  0  0  0  4 


ftOM 

toftrti 

Mm 


«  S  5 


•  u  jo 

I  U  4H 

a  <  J 

i  a.  •  0ft 
N  H  *  I 

«  3  »  ^ 

•  HH> 


to  I  r 

laMs; 

°,7  * . ; 

dh  <  ft 

d  N  .  -F 


1! 

X  M 

Ed 

to  HI 

S3 

t  o  Z 

;si 

ii 


to  4! 

J! 

£2 


O  ft 

to  JS 


ft 

ft 


11-13 


Figure  11-9.  AdaQuest  Coverage  History  Detail  Report 


21-JAM-1M2  12:43  ADAQUEST  COVERAGE  HISTORY  SOMMtY  REPORT 

Library:  ADALEXlNOM 

t  I  Branch**  I  I  Branch**  I  PER  CENT  I 

Proqraa  Unit  Lin*  I  I  Hit  I  In  Unit  I  COVERAGE  I  Llat  of  Branch**  Hot  Hit 
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Figure  11*10.  AdaQuest  Coverage  History  Summary  Report 
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08— MAR— 1993  13:01 


ADAQUEST  TEST  RUM  REPORT 


PAGE  1 


t 


Trace  File 
Time  of  Run 


Description 
#  of  Teat  Cases 


USR :  t  ADATEST  ]  SUN_RUN4  .  ETF  ;  1 
08-MAR-1993  11:35:06 
TIKING_RUN_1 

Sample  run  for  timing  data  using  test  file  1 
1 


Test  Run  Units  : 

LL_COMPILE  :  BOOT 

Instrumented  rrom  File  : 

USR: (ADATEST] LL_COMPILE. ADA; 1 
Instrumentation  Parameters  : 

Timing  -  CPU 

Timing  Intervals  (Start/Stop  Source  Lina  Numbers)  : 
394  -  446 
4S3  -  459 
463  -  496 
653  -  663 
667  -  717 
731  -  733 
737  -  730 


Ui_SUPPORT  ■  BOOT 

Instrumented  From  File  : 

USR> (ADATEST . ADALEX3 ] LL_SUP_BODT . ADA; 1 
Instrumentation  Parameters  i  ” 

Timing  -  CPU 

Timing  intervals  (Start/Stop  source  Line  Numbers)  : 
330  -  383 
391  -  351 
365  -  460 
465  -  501 
508  -  546 


Figure  AdaQuest  Interval  Test  Run  Report 
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08 -MAR-19 92  12:02  ADAQUEST  INTERVAL  TIMING  REPORT  PAGE  1 

Start  /  Stop  Number  of  Minina  Tlae  Maximum  Time  Average  Tlae 
Interval  Line  Number  Executions  hh:ma:ss.cc  hh:mm:ss.cc  hh:ma:ss.cc 


-  TEST  RUN  INFORMATION  - 


Trace  File 
Tine  of  Run 
Name 

Description 
<  Test  Cases 


USR: [ADATEST] SUN_RUN4 . ETF;1 
08— MAR— 1992  11:35:06 
TIMING_RUN_1 

Sample  run  for  tining  data  using  test  file  1 
1 


By  Test  Case 


-  REPORT  QUALIFIERS  - 


;  No 


08-MAR-1992  12:02  ADAQUEST  INTERVAL  TIMING  REPORT  PAGE  2 

Start  /  Stop  Number  of  Minimum  Tine  Maximum  Tine  Average  Tine 
Interval  Line  Number  Executions  hh:mmiss.co  hb:mm:ss.cc  hh:ma:ss.cc 


CPU  TIMING  FOR  COMP  UNIT  :  LL_C0MPILE : BOOT ' 


1 

394  /  446 

64 

00:00:00.00 

00:00:00.01 

00:00:00.00 

2 

453  /  459 

64 

00:00:00.00 

00:00:00.01 

00:00:00.00 

3 

463  /  496 

1 

00:00:00.34 

00:00:00.34 

00:00:00.34 

5 

667  /  717 

1 

00:00:00.43 

00:00:00.43 

00:00:00.43 

6 

721  /  723 

1 

00:00:00.78 

00:00:00.78 

00:00:00.78 

7 

727  /  730 

1 

00:00:00.82 

00:00:00.82 

00:00:00.82 

CPU  TIMING 

FOR  COMP  UNIT  i 

i  LL_SUPPORT : BODY 

4 

465  /  501 

4 

00:00:00.01 

00:00:00.01 

00:00:00.01 

5 

508  /  546 

3 

00:00:00.00 

00:00:00.00 

00:00:00.00 

Figure  11*12.  AdaQuest  Interval  Timing  Report 
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12.  AutoFlow-Ada 

AutoFlow-Ada  generates  flowcharts  from  Ada  source  code.  These  flowcharts  can  be 
used  to  help  understand  an  Ada  program  and  to  document  it.  Versions  of  AutoFlow  that  op¬ 
erate  on  C,  Cobol,  Fortran,  and  Pascal  code  are  also  available.  In  addition  to  flowcharts, 
these  other  versions  generate  structure  charts  and  can  interface  with  the  KnowledgeWare/ 
ADW  and  the  Texas  Instruments  IEW  CASE  systems  via  an  import  file.  The  C  version  also 
includes  the  capability  to  instrument  source  code  to  report  on  test  coverage  at  the  branch 
level;  results  can  then  be  automatically  annotated  on  flow  charts. 


12.1  Tool  Overview 

AutoFlow  was  developed  by  AutoCASE  Technology.  The  AutoFlow  family  as  a  whole 
has  over  3,000  users.  The  first  Ada  version  of  this  product  was  released  early  in  1992  and 
has  over  10  users.  It  runs  on  IBM  PC  machines  under  DOS  (version  3.0  or  higher)  and  OS/ 
2.  The  evaluation  was  performed  on  version  1.02  of  AutoFlow-Ada.  At  the  time  of  evalu¬ 
ation,  the  price  for  AutoFlow  was  $9,950. 

AutoFlow-Ada  generates  self-explanatory  block-structured  flowcharts  using  a  flow¬ 
chart  layout  copyrighted  by  AutoCASE  Technology.  It  is  intended  for  use  on  programs 
with  correct  Ada  syntax,  that  is,  compilable  programs.  Compiler  directives  are  treated  as 
comments  and  not  expanded.  Consequently,  in  some  circumstances,  it  may  be  necessary  to 
use  the  fully  expanded  preprocessed  listing  file  provided  by  many  Ada  compilers  as  input 
to  AutoFlow-Ada.  The  tool  can  be  used  in  interactive  or  batch  mode.  In  interactive  mode  it 
allows  the  user  to  both  create  and  browse  flowcharts,  selectively  saving  or  printing  chosen 
charts.  In  batch  mode,  all  produced  flowcharts  are  automatically  saved  to  disk. 

Usually  one  flowchart  is  generated  for  each  Ada  procedure.  Some  of  these  flowcharts 
may  be  very  large  and  various  options  are  provided  for  dealing  with  flowcharts  that  cannot 
fit  on  a  single  page.  The  best  of  these  is  a  block-structured  page-break  algorithm  that  uses 
a  top-down  refinement  approach  to  break  a  large  flowchart  into  subcharts  that  can  be  pre¬ 
sented  on  separate  pages.  Additional  flexibility  is  provided  by  allowing  the  user  to  specify 
the  size  of  page  used.  Another  option  is  to  limit  the  size  of  the  box  in  which  flowcharts  are 
presented.  In  this  case,  flowchart  elements  that  do  not  fit  into  the  specified  box  are  repre¬ 
sented  by  a  string  of  dots.  Alternatively,  the  user  can  request  that  a  flowchart  saved  to  disk 
is  divided  into  strips  that  can  be  manually  combined  to  make  a  large  chart 
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12.2  Observations 

Ease  of  use.  The  installation  and  operation  of  AutoFlow-Ada  is  straightforward.  The 
tool  is  fast:  the  documentation  cites  an  example  of  generating  flowcharts  for  an  Ada  pro¬ 
gram  in  excess  of  ten  thousand  lines  of  code,  where  the  average  time  to  generate  each  flow¬ 
chart  page  was  less  than  0.5  seconds. 

AutoFlow-Ada  includes  a  number  of  special  options  and  utilities  that  facilitate  its  use. 
The  utility  mkdoall ,  for  example,  generates  command  files  that  will  invoke  AutoFlow-Ada 
on  multiple  source  files.  Utilities  and  functions  that  support  its  use  with  non-IBM  compat¬ 
ible  printers  are  also  provided.  Additionally,  a  file  format  conversion  utility  is  available  to 
convert  ASCII  file  into  PostScript,  HPGL,  and  PIC  formats  that  can  be  sent  to  special  out¬ 
put  devices,  or  used  with  desktop  publishing  software,  to  prepare  high  quality  documenta¬ 
tion. 

Documentation  and  user  support.  The  documentation  is  sufficient  for  tool  use.  Au- 
toCASE  Technology  provided  good  support  and  was  helpful  and  prompt  in  addressing  en¬ 
countered  problems. 

Ada  restrictions.  AutoFlow-Ada  supports  full  Ada,  the  only  restrictions  being  that 
each  procedure  or  function  is  limited  to  2,048  basic  blocks  and  that  input  source  lines  are 
limited  to  127  characters. 

Problems  encountered.  AutoFlow-Ada  ran  on  the  sample  Ada  Lexical  Analyzer  Gen¬ 
erator  source  code.  Various  problems  were  encountered  if  the  size  of  generated  flowcharts 
was  not  constrained  or  when  some  particular  page  sizes  were  specified.  These  problems  in¬ 
cluded  the  process  hanging  and  incorrect  referencing  between  subcharts.  AutoCASE  cor¬ 
rected  the  underlying  problems  and  provided  a  new  copy  of  the  tool. 

12.3  Planned  Additions 

Version  2  of  AutoFlow-Ada  is  scheduled  for  release  in  the  fourth  quarter  of  1992.  It  will 
include  the  generation  of  structure  charts  and  a  graphical  user  interface.  This  version  will 
also  be  available  on  major  Unix  platforms. 
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12.4  Sample  Outputs 

Figures  12-1  through  12-6  provide  sample  outputs  from  AutoFlow-Ada. 
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Figure  12-1.  AutoFlow-Ada  Page  1  of  6  Flowgraph  for  Function  ALTERNATE 
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12*3.  AutoFlow-Ada  Page  3  of  6  Flowgraph  for  Function  ALTERNATE 


Figure  12-4.  AutoFlow-Ada  Page  4  of  6  Flowgraph  for  Function  ALTERNATE 
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Figure  12-5.  AutoFlow-Ada  Page  5  of  6  Flowgraph  for  Function  ALTERNATE 
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Figure  12-6.  AutoFlow-Ada  Page  6  of  6  Flowgraph  for  Function  ALTERNATE 
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13.  DISTRIBUTED  DEFECT  TRACKING  SYSTEM  (DDTs) 

The  Distributed  Defect  Tracking  System  (DDTs)  provides  for  tracking  and  managing 
defects  and  change  requests  throughout  the  life  cycle  of  a  software  or  hardware  product.  It 
is  designed  to  support  large  organizations  with  multiple  sites  and  so  is  fully  distributed  and 
suitable  for  use  in  a  heterogeneous  network.  DDTs  supports  multiple  development  teams, 
allowing  data  to  be  maintained  for  several  projects  simultaneously.  In  addition  to  reporting, 
searching,  and  query  tools,  DDTs  informs  appropriate  users  of  changes  to  defect  states  to 
provide  closed-loop  tracking. 


13.1  Tool  Overview 

This  product  was  developed  by  QualTrak  Corporation  and  has  been  marketed  since 
1989.  There  are  over  100  sites  using  DDTs,  with  some  estimated  5,000  users.  QualTrak 
provides  consultancy  and  training,  and  supports  its  product  with  a  hot-line  service  and  an 
on-line  users  group.  A  newsletter  is  expected  to  become  available  in  the  near  future.  DDTs 
is  available  under  SunOS  on  Sun-3  and  Sun-4  systems,  under  HP-UX  on  HP-9000,  under 
ADC  on  IBM  RS-6000  and  Apollo  systems,  under  Ultrix  on  DECstations  and  VAXs,  and 
under  SCO  Unix.  It  uses  troff,  tbl ,  sort ,  ctwk  Bourne  shell  Unix  utilities,  but  is  DBMS  inde¬ 
pendent.  For  a  local  network,  it  supports  network  file  sharing  (NFS),  ethemet  client/server 
User  Datagram  Protocol  (UDP),  and  the  Transmission  Control  Protocol  (TCP).  Electronic 
mail  is  supported  for  remote  networks.  The  examination  was  performed  on  version  2. 1 .6  of 
this  product  running  on  a  Sun-3  system.  At  the  time  of  evaluation,  prices  for  DDTs  started 
at  $6,000. 

DDTs  groups  defects  by  project  to  allow  reporting  on  both  the  defects  in  a  particular 
project  and  to  support  organizational  quality  assurance  activities  across  projects.  Each 
project  is  associated  with  one  computer  system,  known  as  the  home  system.  All  of  a 
project’s  defects  reside  on  that  home  system  and  on  the  submitters’  systems  as  well.  In  ad¬ 
dition,  a  subscription  facility  to  a  project  is  supported;  in  this  case,  defects  are  maintained 
on  the  home  system  and  the  subscriber’s  system.  A  secure-in  dial  facility  is  available  that 
provides  easy  local  access  to  defect  information  about  remote  projects.  Closed-loop  track¬ 
ing  means  that  defect  submitters  are  automatically  informed  of  all  changes  in  a  defect’s  sta¬ 
tus  by  electronic  mail. 
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DDTs  can  be  used  in  either  a  menu-driven  or  command-driven  manner.  In  the  first  case, 
a  user  has  two  avenues  of  access;  one  provides  the  full  set  of  functions  suitable  for  a  devel¬ 
oper,  and  the  other  provides  a  subset  of  functions  tailored  towards  defect  submitters. 

The  system  defines  a  defect  life  cycle  which  allows  defects  to  be  managed  using  a  state 
transition  mechanism;  both  forward  and  backward  transitions  are  supported.  A  defect  life 
cycle  starts  with  its  submission  and,  usually,  ends  with  its  resolution.  There  are  nine  pre¬ 
defined  defect  states,  though  the  user  can  define  others  by  including  them  in  a  state  transi¬ 
tion  table  and  defining  allowable  state  transitions.  (DDTs  warns  of  any  illegal  transition 
attempts.)  Defects  can  be  classified  as  enhancement  requests  and  subsequently  tracked  by 
DDTs. 

DDTs  uses  a  template  to  guide  user  entry  of  defect  reports.  Information  is  grouped  into 
the  following  areas:  detection,  submitter,  laboratory,  resolution,  and  verification  informa¬ 
tion.  Detection  information  is  used  to  specify,  for  example,  the  detection  method,  the  de¬ 
velopment  phase  in  which  the  defect  was  detected,  and  defect  severity  (one  of  five  levels) 
in  addition  to  identification  of  the  test  system  operating  system  and  affected  project.  When 
available,  information  about  the  defect  submitter  is  added  automatically.  The  laboratory  in¬ 
formation  captures  information  pertaining  to  diagnosing  the  defect.  In  addition  to  identify¬ 
ing  the  responsible  engineer,  it  records  the  type  and  cause  of  the  underlying  defect, 
recommended  change,  and  estimated  fix  time  and  date.  The  resolution  information  is  sim¬ 
ilar.  Again  the  responsible  person  is  identified,  but  this  time  the  actual  effort  required  to 
make  the  fix,  the  development  phase  when  the  fix  was  made,  and  location  of  actual  changes 
are  recorded.  Finally,  the  verification  information  identifies  who  accepted  the  resolution. 

Defect  reports  can  be  supplemented  by  enclosures.  These  are  additional  files  containing 
supplemental  ASCII  text.  They  can  be  used,  for  example,  to  include  the  data  files  needed 
to  reproduce  a  problem.  There  is  no  limit  to  the  number  of  enclosures  that  can  be  linked  to 
a  defect  report.  DDTs  automatically  brings  up  a  change  editor  for  creating  enclosures.  Al¬ 
though  the  v/  editor  is  used  by  default,  the  user  can  request  other  editors. 

DDTs  provides  several  predefined  report  formats.  These  conform  with  the  proposed 
IEEE  Standard  P-1044,  and  with  DoD-STD-2 1 67  A.  They  include,  for  example,  a  list  of  all 
unresolved  defects  for  one  or  more  projects  and  a  list  of  defects  in  selected  states.  A  number 
of  sorting  filters  are  available  for  use  in  constructing  specialized  report  formats.  A  nice  fea¬ 
ture  is  a  weekly  report  program  that  can  be  used  to  produce  reports  automatically.  The  met¬ 
rics  provided  in  weekly  reports  include  such  information  as  the  arrival  rate,  fix  rate,  number 
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defects  assigned  to  each  project  engineer,  resolved  and  unresolved  defects,  and  when  these 
defects  were  found  and/or  fixed. 

On-line  defect  report  displays  are  also  available.  These  allow  a  user  to  identify  all  the 
defects  he  submitted  or  the  unresolved  defects  he,  or  another  engineer,  is  responsible  for. 
The  contents  of  selected  defect  reports  can  be  displayed  with  an  index  pointer  used  to  move 
between  different  defect  reports.  The  user  can  also  search  this  index  for  a  given  string.  Ad¬ 
ditional  search  and  query  facilities  are  provided  to  answer  ad  hoc  questions.  The  search  op¬ 
tion  allows  displaying  all  unresolved  defects  and  unresolved  defects  of  severity  1  and  2  for 
one  or  more  projects.  It  matches  a  user-defined  string  against  the  one-line  summaries  of  de¬ 
fect  descriptions  kept  for  each  defect.  The  query  function  allows  the  user  to  specify  a  search 
string  composed  of  defect  keywords,  operators,  and  values  combined  in  a  C-like  expres¬ 
sion. 

DDTs  provides  explicit  support  for  a  number  of  administration  functions.  These  in¬ 
clude  cleaning  up  log  files,  checking  and  repairing  the  database,  showing  the  status  of 
DDTs  projects,  and  managing  projects.  Setting  up  a  new  project  involves  setting  applicable 
template  files  and  state  transition  rules.  The  administrator  also  specifies  the  individuals  and 
groups  who  should  be  notified  of  changes  to  defect  states  and  those  who  are  permitted  to 
change  defect  states.  He  can  customize  DDTs  by  adding  or  deleting  a  defect  state,  adding 
or  deleting  a  field  in  the  defect  reports,  and  changing  the  dialog  that  occurs  with  the  user 
when  a  state  transition  occurs.  In  addition  to  modifying  the  predefined  management  re¬ 
ports,  the  administrator  can  create  new  report  types.  Finally,  the  administrator  is  provided 
with  guidance  for  converting  existing  defect  reports  to  DDTs  format 


13.2  Observations 

Ease  of  use.  DDTs  recognizes  two  types  of  users:  defect  submitters,  and  developers 
who  repair  defects.  While  the  menu  interface  provided  for  each  type  of  user  is  similar,  this 
distinction  allows  providing  a  simpler  interface  for  defect  submitters.  Context-sensitive 
help  is  available  in  both  cases.  Additional  guidance  for  expert  users  is  available  as  a  set  of 
tips.  These  take  the  form  of  short  excerpts  from  the  on-line  manual  pages  and  provide  an 
introduction  to  the  search  and  query  functions.  Expert  users  can  also  use  DDTs  through  a 
command  interface. 

Template  file  mechanisms  provide  for  customization  and  specialized  defect  reports  can 
be  defined  to  augment  the  predefined  reports  provided  by  DDTs.  The  system  includes  sev- 
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eral  levels  of  flexibility.  For  example,  each  project  can  employ  different  screens,  prompt¬ 
ing,  and  states  transitions. 

The  DDTs  import  facility  is  a  valuable  capability.  It  allows  the  definition  of  converters 
that  take  existing  defect  reports  in  a  defined  format,  convert  them  to  a  defined  DDTs  for¬ 
mat,  and  place  them  in  the  DDTs  database. 

Documentation  and  user  support.  DDTs  is  designed  so  that  it  can  be  used  without 
documentation.  Nevertheless,  it  is  well  supported  by  documentation  that  includes  a  tutorial, 
several  examples,  and  sample  outputs.  Unix-like  on-line  manual  pages  provide  for  quick 
reference  and  can  be  integrated  into  the  on-line  manuals  supported  by  a  Unix  system. 

Installation  procedures  are  well  described.  They  include  special  information  that,  for 
example,  helps  a  system  administrator  determine  where  to  place  the  product  by  providing 
an  estimate  of  the  rate  of  growth  of  the  database,  as  well  as  estimates  of  dynamic  storage 
requirements. 

Problems  encountered.  No  problems  were  encountered  in  the  use  of  DDTs. 


13.3  Recent  Changes  and  Planned  Additions 

A  new  product.  Remote  Distributed  Defect  Tracking  System  (RDDTs),  released  in 
summer  1992,  provides  a  restricted  submit-only  version  of  DDTs. 

DDTs  release  3.0  is  due  to  be  released  in  December  1992.  This  version  will  support  an 
X-l  1  graphical  user  interface  as  well  as  the  existing  tty  interface.  It  will  also  support  Post¬ 
Script  for  enhanced  graphical  charts. 

The  QTET  test  harness  is  a  product  under  development  to  provide  an  interface  between 
DDTs  and  test  execution  tools.  It  is  based  on  the  public  domain  Test  Environment  Toolkit; 
QualTrak  Corp.  has  added  a  graphical  user  interface  and  bound  the  Test  Environment  Tool¬ 
kit  to  DDTs.  QTET  is  expected  to  become  available  in  the  second  quarter  of  1993. 

13.4  Sample  Outputs 

Figures  13-1  through  13-7  provide  sample  outputs  from  DDTs.  Figure  13-8  provides  an 
example  of  the  outputs  available  with  the  DDTs  graphical  user  interface;  it  was  supplied  by 
QualTrak  Corp. 
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Bug  SFDaa03277  DDTs  Submitted  910305 

ASSIGNED  defect  report  bugs(l),  version  2.1  Assigned  910305 

2  enclosures 

'Enclosure  date  stamp  is  incorrectly  updated* 

DETECTION  INFORMATION 
Detection  method:  customer  use 
Detected  in  phase:  post-release 
Test  program  name:  bugs 
Test  system: 

Version  of  OS: 

Problem  severity :  3 

Affects  project ddts 
Need  fix  by:  910909 

SUBMITTER  INFORMATION 
Submitter :  Mike  Manley 

Organization:  QT  LABX 
Phone  number:  33137 
Address :  mikey 1 mmanley 

••••*«*•••••  Problem  (Added  910305  by  munley)  **••***««•» 

From  Lori  Pope  at  Pacesetter 

>2.  (New?)  when  we  attempt  to  modify  an  existent  enclosure  by: 

>  1.  selecting  *m*  when  viewing  the  enclosure 

>  2.  exit  the  editor  without  saving  the  modifications  (eg.  no 

>  modifications  were  performed)  -  in  vi  you  would  exit  with  *:q!* 

>  3.  Exit  ddts  by  typing  *q*. 

>the  enclosure's  modification  date  is  updated. 

» 

>For  now,  the  work  around  is  to  exit  ddts  by  typing  *x*  Instead  of 

> *q"  -  this  exits  ddts  without  saving  any  changes. 

>That  may  not  be  satisfactory  if  you  hava  made  changes  in  other  SHRs 

>that  you  wish  to  keep. 


•*•*•*••***•  design  ideas  (Added  910606  by  ddts)  •*•*••***** 

stat  the  file  before  going  to  the  editor  and  see  if  the  time  changed 


Figure  13-1.  DDTs  Sample  Defect  Report 


LABORATORY  INFORMATION 
Assigned  engineer:  rico 
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DDTS  MANAGEMENT  SUMMARY 
of 


DEFECTS  by  PROJECT  by  STATE 
(Tue  Jul  14  14:19:11  EDT  1992) 


Project 

New 

Assnd 

Open  Rslvd 

Verlf 

Dup 

POBtp 

Total 

DDTs 

1 

18 

0 

84 

0 

0 

0 

103 

TOTAL 

1 

18 

0 

84 

0 

0 

0 

103 

Youngest  Bug  Date 

-> 

911221 

• 

Oldest  Bug  Date 

-> 

901029 

Software  Versions 

-> 

2.1  3.0  2.1.3 

unk  2.2  2.0.3 

2.1.2  2.0  2.0.1 
bar  1.0 

O.S.  Versions 

-> 

4.1  4.0  Sun 

SunOS  unk  a 
none  any  3 . 5 

« 

DOTS  MANAGEMENT  SUMMARY 
Of 

DEFECTS  by  PROJECT  by  SEVERITY 
RESOLVED  t  UNRESOLVED  BUGS 
(Tue  Jul  14  14:19:16  EDT  1992) 


Project ' 

Sevl 

Sevl 

Sev3 

Sev4 

Sev5 

Total 

DDTs 

10 

14 

65 

12 

2 

103 

TOTAL 

10 

14 

65 

12 

2 

103 

Youngest  Bug  Date  -> 

911221 

Oldest  Bug  Date  *■> 

901029 

Software  Versions  “> 

2.1  3.0  2.1.3 

unk  2.2  2.0.3 

2.1.2  2.0  2.0.1 
bar  1.0 

• 

O.S.  Versions  -> 

4.1  4.0  Sun 

SunOS  unk  a 
none  any  3 . 5 

Figure  13-2.  DDTs  Management  Summary  Report:  Defect  Reports 
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DOTS  MANAGEMENT  SUMMARY 
of 

DEFECTS  by  ENGINEER  by  SEVERITY 
UNRESOLVED  DEFECTS  ONLY 
(Tu«  Jul  14  14:19:22  EOT  1992) 


Assigned 

Engineer 

Sav 

1 

Sev 

2 

f n 

w  ? 

Sev 

4 

Sev 

5 

Total 

■aanley 

0 

0 

4 

0 

0 

4 

rioo 

0 

1 

9 

4 

1 

15 

davld 

0 

0 

0 

0 

0 

0 

davep 

0 

0 

0 

0 

0 

0 

carol 

0 

0 

0 

0 

0 

0 

UNASSIGNED 

0 

0 

0 

0 

0 

0 

TOTAL 

0 

1 

13 

4 

1 

19 

Projects  surveyed  -> 
Youngest  Bug  Date  •> 
Oldest  Bug  Date  -> 
Software  Versions  -> 


O.S.  Versions  -> 


DOTS 

911221 

901029 

2.1  3.0  2.1.3 

unk  2.2  2.0.3 

2.1.2  2.0  2.0.1 
bar  1.0 

4.1  4.0  Sun 

SunOS  unk  a 
none  any  3 . 5 


Flgurel3-2  continued:  DDTs  Management  Summary  Report:  Defect  Reports 
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DOTS  MANAGEMENT  SUMMARY 
of 

DEFECTS  by  SUBMITTING  ENGINEER  by  SEVERITY 
(Tua  Jul  14  14:19:35  EDT  1992) 


Subaitting 

Sav 

Sav 

Sav 

Sav 

Sav 

Total 

Enginaar 

1 

2 

3 

4 

5 

■aanley 

4 

3 

29 

3 

1 

40 

cindy 

3 

5 

22 

7 

1 

38 

ddta 

2 

6 

11' 

2 

0 

21 

carol 

0 

0 

3 

0 

0 

3 

rico 

1 

0 

0 

0 

0 

1 

UNASSIGNED 

0 

0 

0 

0 

0 

0 

TOTAL 

10 

14 

65 

12 

2 

103 

Projects  surveyed  -> 
Youngest  Bug  Data  “> 
01 das t  Bug  Data  «> 
Softwara  Vara Iona  -> 


O.S.  Varalona  -> 


DDTs 

911221 

901029 

2.1  3.0  2.1.3 

uok  2.2  2.0.3 

2.1.2  2.0  2.0.1 

bar  1.0 

4.1  4.0  Sub 

SunOS  uak  a 
nona  any  3 . 5 


Figurei3>2  continued:  DDTs  Management  Summary  Report:  Defect  Reports 


\ 

PART  II 

DDTs 

> 

DOTS  MANAGEMENT  SUMMARY 

DEFECT  ARRIVAL  6  REPAIR  RATE 

1 

ALL  SEVERITY  LEVELS 

(Tue  Jul  14  14*19:40  EDT  1992) 

Hack 

Date  f  New  t  Resolved 

Dlff 

4  Unresolved 

1 

901028  1  0 

1 

2 

2 

901104  0  0 

0 

2 

3 

901111  3  2 

1 

3 

1 

4 

901118  0  0 

0 

3 

5 

901125  0  0 

0 

3 

6 

901202  0  0 

0 

3 

7 

901209  3  1 

1 

4 

a 

901216  1  0 

1 

5 

9 

901223  0  0 

0 

5 

10 

901230  0  0 

0 

5 

1 

11 

910106  1  0 

1 

6 

12 

910113  4  0 

4 

10 

13 

910120  1  0 

1 

11 

14 

910127  4  0 

4 

15 

15 

910203  0  0 

0 

15 

16 

910210  1  0 

1 

16 

17 

910217  0  0 

0 

16 

18 

910224  0  0 

0 

16 

» 

19 

910303  6  0 

6 

22 

20 

910310  0  0 

0 

22 

21 

910317  1  0 

1 

23 

22 

910324  2  10 

-8 

15 

23 

910331  0  0 

0 

15 

59 

•  •  • 

911208  3  2 

1 
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Figure  13-3.  DOTS  Management  Summary  Report:  Defect  Arrival  and  Repair  Rate  (All  Levels) 
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Figure  13-4.  DDTs  Management  Summary  Report:  Detect  Arrival  and  Repair  Rate  (Sev.  i  &  2) 
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Figure  13-5.  DDTs  Management  Summary  Report:  Sample  Histograms 
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DOTS  management  SUW4ARY  # 

Three  Lin*  Bug  Summaries 
Tue  Jul  14  14:30:03  EM  1993 


DETECTS  FOR  PROJECT  DM* 

Bug  Number  -  000**00051,  Project  “  DM*, 

St-N,  Sv-3 ,  Thing*  to  remember  for  th*  DM*  Installation  Upgrade 
Module:  upgrade,  Vers  -  3.1,Engr  »  mmanley, Found:  911019, Fixed:  ?? 

Bug  dumber  -  000*a00079,  Project  -  DM*, 

St- A,  Sv-2,  bugs(l)  index  printing  need*  to  be  much  faster 
Module:  bug*{l),  ver*  -  3.l,Engr  -  rioo, round:  911313, Fixed:  ?? 

Bug  Number  -  000**00036,  Project  -  DM*, 

St-  A,  Sv-3 ,  Only  1  line  should  be  repeated  on  page  forward  thru  index 
Module:  bugs(l).  Vers  -  3.1, Engr  -  rico, Found:  91090B, Fixed:  7? 

Bug  Number  -  000**00075,  Project  -  DMs, 

St-  A,  Sv— 3 ,  This  is  a  reminder  about  m*lloc(3) 

Module:  bugs(l).  Vers  -  3.0, Engr  -  rico, Found:  911301, Fixed:  7 7 

Bug  MUmber  “  000**00076,  project  -  DMs, 

St-A,  Sv-3,  Adminbug  needs  to  set  up  CM  stuff 

Nodule:  adminbug,  Vers  -  3.1, Engr  -  mmanley, Found:  911313, Fixed:  77 
Bug  Number  -  QQOaaOOOBl,  Project  -  DDT*, 

St-A,  Sv-3,  the  mail. subject  template  file  needs  documentation 
Module:  mail. subject.  Vers  -  3.0, Engr  -  rico, Found:  911313, Fixed:  77 

Bug  Number  -  QQQaa00086,  Project  -  DMs, 

St-A,  Sv-3,  CM- notify  needs  to  be  in  proj. notify  file 

Module:  adminbug.  Vers  -  3.0, Engr  -  mmanley, Pound:  911319, Fixed:  7? 

Bug  Number  -  SFDaa03365,  Project  -  DMs, 

St-A,  Sv-3,  C.E.  suggests  moving  A  per- proj acting  some  template  files 
Module:  faugs(l),  Vers  -  3.1, Engr  -  rioo, Found:  910114, Fixed:  77 

Bug  Number  -  SFD*a03370,  Project  -  DMs, 

St-A,  Sv-3,  6.E.  wants  to  have  a  mechanism  for  total  mail  supression  per  sta 

Module:  bugmail,  Vers  -  3.1, Engr  -  rioo, Found:  910130, Fixed:  77 

Bug  Number  -  SPD*a03376,  Project  -  DMs, 

St-A,  Sv-3,  Last-mod  not  updated  when  enclosure  is  modified 
Module:  bugs(l).  Vers  -  3.1, Engr  -  rioo, Found:  910305, Fixed:  77 


Bug  Number  -  8FD**03390,  Project  -  DMs, 

St-R,  Sv-5,  Mew  bugs  loaded  via  bbox  are  not  displayed 

Module:  bugs(l),  Vers  -  3.1, Engr  -  rioo, Found:  910419, Fixed:  911331 


Figure  13-6.  DDTs  Management  Summary  Report:  Bug  Summaries 
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DOTS  MANAGEMENT  SUMMARY 
Of 

General  statistics 
(Tua  Jul  14  14:30 >09  EOT  1993) 


Projects  surveyed 

-> 

DOTS 

1 

Youngest  Bug  Data 

-> 

911331 

Oldest  Bug  Date 

-> 

901039 

Software  Versions 

-> 

3.1  3.0  3.1.3 

unk  3.3  3.0.3 

3.1.3  3.0  3.0 

bar  1.0 

1 

o.s.  versions 

-> 

4.1  4.0  Sun 

SunOS  unk  a 
none  any  3 .  S 

Assigned  Engineer  Statistics 
Of  103  assigned  bugs: 


NO  OR 

assigned 

0  bugs  -> 

0.00* 

snanley 

assigned 

44  bugs  -> 

43.73* 

rico 

assigned 

43  bugs  •> 

41.75* 

david 

assigned 

IS  bugs  -> 

14. 56* 

carol 

assigned 

1  bugs  -> 

0.97* 

Bug  Submission  Statistics 


103  bugs 

submitted: 

■sanley 

submitted 

40  bugs  “> 

38.83* 

cindy 

submitted 

38  bugs  -> 

36.89* 

ddts 

submitted 

31  bugs  “> 

30.39* 

carol 

submitted 

3  bugs  -> 

3.91* 

rloo 

submitted 

1  bugs  •*> 

0.97* 

Bow  round  Statistics 
Of  103  bugs  found: 


10  bugs 

found  by 

author  ooda  review  t 

-> 

9.71* 

30  bugs 

found  by 

in-house  normal  use 

-> 

19.43* 

3  bugs 

found  by 

group  code  review 

»> 

1.94* 

60  bugs 

found  by 

customer  use 

»> 

58.35* 

8  bugs 

found  by 

interactive  test 

-> 

7.77* 

1  bugs 

found  by 

random  unplanned  test 

■> 

0.97* 

3  bugs 

found  by 

functional  test 

«> 

1.94* 

Bear  Resolved  Statistics 
Of 


84  bugs  resolved: 

53  bugs  resolved  by 

source  code 

-> 

63.10* 

4  bugs  resolved  by 

design 

-> 

4.76* 

6  bugs  resolved  by 

documentation 

-> 

7.14* 

13  bugs  resolved  by 

no  fix 

■> 

15.48* 

3  bugs  resolved  by 

unreprodualble 

-> 

3.57* 

5  bugs  resolved  by 

not  a  bug 

-> 

5.95* 

When  Caused  Statistics 


» 


Figure  13-7.  ODTs  Management  Summary  Report:  General  Statistics 
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81  bugs: 

15  bugs  caused  during 

design 

-> 

18.53* 

36  bugs  caused  during 

post-release 

-> 

44.44* 

13  bugs  caused  during 

alpha  test 

-> 

14.81* 

3  bugs  caused  during 

beta  test 

-> 

3.70* 

10  bugs  caused  during 

implementation 

-> 

13.35* 

3  bugs  caused  during 

investigation 

-> 

3.70* 

3  bug*  caused  during 

integration 

-> 

3.47* 

found  Statistics 

103  bugs  found: 

13  bugs  found  during 

alpha  test 

•> 

11.65* 

74  bug*  found  during 

post-release 

-> 

71.84* 

4  bugs  found  during 

implementation' 

-> 

3.88* 

3  bugs  found  during 

design 

-> 

1.94* 

9  bugs  found  during 

integration 

-> 

8.74* 

1  bugs  found  during 

investigation 

-> 

0.97* 

1  bugs  found  during 

functional  test 

-> 

0.97* 

When  find  Statistic* 
Of  SO  bugs  fixed i 
51  bugs 
13  bugs 
3  bugs 
3  bugs 
3  bugs 
6  bugs 


fixed  during 
fixed  during 
fixed  during 
fixed  during 
fixed  during 
fixed  during 


post-release 
alpha  teat 
beta  test 

integration 

design 

investigation 


-> 

-> 

-> 

-> 

-> 

-> 


63.75* 

16.35* 

3.50* 

3.75* 

3.75* 

7.50* 


3  bugs  fixed  during  imple 

mentation 

->  3 

Severity  Statistics 

Number  of  severity  1  bugs 

m 

10 

-> 

9.71* 

Number  of  severity  3  bug* 

m 

14 

-> 

13.59* 

Number  of  severity  3  bugs 

■ 

65 

-> 

63.11* 

Number  of  severity  4  bugs 

am 

13 

-> 

11.65* 

Number  of  severity  5  bugs 

am 

3 

-> 

1.94* 

Status  Statistics 

Number  of  nee  bugs 

m 

1 

-> 

0.97* 

Number  of  open  bugs 

m 

0 

-> 

0.00* 

Number  of  resolved  bugs 

m 

84 

■> 

81.55* 

Number  of  postponed  bugs 

am 

0 

-> 

0.00* 

Number  of  duplicate  hugs 

w 

0 

-> 

0.00* 

Number  of  verified  bugs 

m 

0 

-> 

0.00* 

Number  of  assigned  bugs 

m 

18 

-> 

17.48* 

Number  of  integrated  bugs 

m 

0 

-> 

0.00* 

Number  of  released  bugs 

m 

0 

-> 

0.00* 

Figure!  3-7  continued:  DDTs  Management  Summary  Report:  General  Statistics 
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Figure  B.  Examples  of  GUI  Outputs 
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EDSA  is  a  browser  that  supports  understanding  and  static  analysis  of  Ada  source  code. 
It  provides  such  capabilities  as  control  and  data  flow  browsing,  pretty-printing,  elision- 
based  viewing,  and  search  management.  In  addition,  its  annotation  capability  can  support 
the  conduct  of  code  reviews  and  inspections,  as  well  as  capture  the  progress  of  formal  ver¬ 
ification  activities. 


14.1  Tool  Overview 

This  product  was  developed  by  Array  Systems  Computing,  Inc.  and  has  been  marketed 
since  1991.  It  has  between  5  and  10  users.  Array  Systems  Computing  provides  software 
consultancy  and  training,  and  performs  independent  verification  and  validation  activities. 
Tool  users  are  supported  by  a  newsletter  and  hot-line  support.  EDSA  is  available  under 
Unix,  VMS,  and  DOS.  An  X-Windows  version  is  available.  The  examination  was  per¬ 
formed  on  version  2.0  of  this  product  running  UNIX  on  a  Sun-4  system  with  OpenWin- 
dows.  At  the  time  of  evaluation,  prices  for  EDSA  started  at  $3,750. 

Use  of  EDSA  starts  with  parsing  an  Ada  source  code  file.  This  produces  an  attributed 
syntax  tree  and  symbol  table  that  are  stored  in  the  user  library.  A  successful  parse  is  not 
required  for  browsing  and  any  errors  encountered  during  parsing  are  reported,  together  with 
appropriate  warnings.  When  a  program  is  contained  in  several  files,  compilation  units  must 
be  parsed  in  compilation  order.  The  output  of  the  parser  is  used  for  browsing  and  EDSA 
can  be  invoked  on  any  of  the  parsed  files  independently.  If  required,  preparation  of  a  pretty- 
printed  output  version  of  the  original  source  file  is  available.  This  uses  standardized  inden¬ 
tation  based  on  the  parse  tree  to  emphasize  the  control  structures  of  the  code. 

EDSA  supplements  the  traditional  random  traversal  and  string  searching  common  to 
many  browsers  and  editors  with  several  logic-based  traversal  methods.  These  additional 
traversal  methods  allow  a  user  to  exploit  the  structure  and  meaning  of  the  code.  Specifical¬ 
ly,  the  following  types  of  traversal  are  supported: 

•  Random.  Movement  is  achieved  by  usage  of  the  cursor  and  scrolling  keys,  and  by  a 
set  of  defined  focus  commands. 

•  String  searching.  String  commands  find  specified  items  in  the  code,  for  example,  the 
statements  that  a  particular  statement  depends  upon. 

•  Syntax  directed.  This  allows  the  user  to  follow  paths  defined  by  the  syntax  tree. 
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•  Dependency.  Provides  for  tracing  back  all  the  statements  that  the  currently  selected 
statement  depends  on  (typically  these  statements  are  the  controlling  statement  and  the 
statements  that  define  its  input  variable  values). 

•  Data  flow.  Allows  tracing  back  to  where  a  variable  was  originally  given  a  value  and 
then  ahead  to  every  usage  of  that  value  until  it  is  changed. 

•  Control  flow.  Follows  the  control  logic  of  the  source  code. 

•  Object-usage.  Allows  visiting  every  statement  where  an  Ada  object  (that  is,  a  vari¬ 
able,  parameter,  component,  or  slice)  appears  in  a  specific  context 

In  each  case,  a  stack  and  backtracking  facilities  are  provided  for  switching  between  paths 
when  more  than  one  path  can  be  followed. 

When  browsing,  the  source  text  is  pretty-printed  in  the  view  window.  This  window 
changes  as  the  file  is  a  traversed  or  new  views  constructed.  The  message  window  displays 
the  most  recent  commands  entered  and,  sometimes,  messages  relating  to  the  current  com¬ 
mand.  The  response  box  is  a  temporary  window  used  to  notify  the  user  of  problems  with 
an  entered  command  or  to  ask  for  verification  of  a  command.  During  browsing,  the  user 
can  switch  to  an  editor  and,  at  the  end  of  the  edit,  cause  the  syntax  tree  and  symbol  table  to 
be  appropriately  updated. 

Views  are  provided  to  help  mitigate  the  complexity  of  perusing  large  programs.  Views 
are  a  selection  of  some  or  all  of  the  statements  in  the  source  code.  By  showing  only  specific 
parts  of  the  code,  they  allow  a  user  to  restrict  himself  to  only  those  features  of  interest,  for 
example,  those  portions  of  the  code  that  are  within  a  particular  depth,  that  use  a  specified 
symbol,  or  that  use  a  specified  structure.  Views  can  be  created,  modified,  printed,  and  com¬ 
bined. 

EDSA’s  statement  annotations  are  useful  for  adding  documentation  to  source  code 
without  modifying  the  original  source  file.  Whereas  comments  exist  in  both  the  source  code 
and  syntax  tree,  annotations  exist  only  in  the  syntax  tree  (although  options  to  cause  them  to 
be  included  in  the  source  code  are  provided).  This  can  be  useful  for  recording  temporary 
observations  during  an  analysis  session  or  for  recording  other  types  of  working  notes.  De¬ 
pending  on  the  value  of  a  customization  parameter,  these  annotations  act  like  special  com¬ 
ments  or  are  hidden  from  view  until  required.  After  editing,  EDSA  can  cause  the  syntax 
tree  and  symbol  table  to  be  appropriately  updated  and  annotations  inherited  from  the  old 
tree,  adjusted  if  necessary,  to  conform  to  the  changes. 

Pebbling  is  another  type  of  annotation.  Here  the  annotations,  or  pebbles,  are  used  to 
record  the  fact  that  statements  have  been  examined  and  that  some  conclusion  about  their 
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correctness  has  been  reached.  The  pebbling  feature  uses  dependency  information  to  link 
each  of  a  statement’s  inputs  to  all  of  the  statements  that  might  provide  values  to  those  in¬ 
puts.  It  propagates  correctness  information  by  automating  a  generalization  of  the  following 
rule  from  propositional  logic:  Given  that  A  is  true,  and  that  A  being  true  implies  that  B  is 
true,  then  it  follows  that  B  must  be  true.  (A,  A  — »  B  =»  B).  The  user  places  white  pebbles 
to  indicate  that  the  statement  and  its  contributors  are  assumed  to  be  correct,  that  is,  globally 
correct,  for  verification  puiposes.  He  places  grey  pebbles  to  indicate  that  only  the  statement 
itself  is  assumed  to  be  correct,  that  is,  locally  correct.  If  the  contributors  to  a  locally  correct 
statement  are  globally  correct,  EDSA  automatically  replaces  a  grey  pebble  with  a  black 
pebble  to  indicate  that  global  correctness  has  been  derived,  although  not  asserted. 


14.2  Observations 

Ease  of  use.  A  user  can  interact  with  EDSA  using  a  command  line  with  auto-comple¬ 
tion,  cursor  keys  or  a  mouse  to  move  around  the  menus  and  command  line,  or  key  bindings. 
In  the  latter  case,  default  keys  are  bound  to  the  most  commonly  used  EDSA  commands;  the 
user  can  adjust  these  bindings  to  customize  EDSA  as  desired.  Additional  opportunities  for 
customization  allow  modifying  text  appearance  and  system  parameters.  Examples  of  sys¬ 
tem  parameters  include  switches  that  specify  whether  annotations  should  be  hidden  and 
whether  the  user  should  be  queried  for  backtracking  to  previously  skipped  paths.  Expertise 
level  is  another  system  parameter.  It  allows  a  user  to  be  assigned  one  of  six  levels  of  exper¬ 
tise  that  are  used  to  determine  the  extent  of  help,  menus,  and  warnings  messages  provided. 

Documentation  and  user  support.  The  documentation  is  extensive  and  includes  sev¬ 
eral  useful  examples.  Array  Systems  Computing  were  prompt  and  helpful  in  responding  to 
queries. 

Problems  encountered.  EDSA  performed  exactly  as  described  in  the  documentation. 
No  problems  were  encountered  during  its  use. 


14.3  Sample  Outputs 

Figures  14-1  through  14-6  provide  sample  outputs  from  EDSA. 
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aeparata  (  Ll_Conpile  ) 
package  body  LL_TOKENS  ia 


procedure  Advance (  aos  :  our  BOOLEAN;  next  out  LLTOKEN ;  sore  in  BOOLEAN 
: —TRUE  )  ia 

procedure  Get_Char (  char  :  out  CHARACTER  )  ia 
begin 

if  End_Of_Eile(  Standard_Input  )  then 

elaif  End_Of_Line(  standard_lpput  )  then 
Sklp_Llne(  Standard_Input  ) / 

eiae 

Get (  Standard_I nput ,  char  ); 
end  if; 
end  Get_Char; 


end  Next^String; 


begin 


—  Skip  white  space  and  coanenta 

while  (  current_char~ ASCII . EXE  )  or  (  current_char-ASCII . HT  )  or  ( 
currentchar- '  '  )  or  (  eurrent_ohar-'-'  )  loop 
if  current_char“ ' - '  then 
Look_Ahead; 

Skip_Line(  Standard_I nput  ); 

end  if; 

Char_Advanoe; 
end  loop; 

if  current_char- ASCII . EOT  then 

elaif  ourrent_char- ' " '  then 
Next_Stxing; 

elaif  current_ohar- ' ' '  then 
Next_Character ; 

elsif  (  current _char  in  UPPEH_CA5E_X>ETTZX  )  or  (  current_ohar  in 
UMSR_CASE_LrtTZX  )  then 
Mext_ldentif ier ; 

else 

Next_Spec_Sy» ; 
end  if; 


end  Advance; 
end  IX  TOKENS ; 


Figure  14*1.  EDSA  Threads  View  of  Compilation  Unit  LL_TOKENS 
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separate  (  Ll_Coapile  ) 
package  body  LL_TOKENS  is 


procedure  Advance (  eos  t  out  BOOLEAN;  next  :  out  LLTOKEN;  sore  :  in  BOOLEAN 
: “TRUE  )  it 


procedure  Next_String  la 
begin 

while  current_char/- ' * '  loop 

exit  when  End_Of_Line(  Standard_Xnput  ); 
end  loop/ 
end  Next_String; 
begin 

—  Skip  white  space  and  c assents 

while  (  current_ch*r-ASCII .  FIX  )  or  (  current_char- ASCII . HT  )  or  ( 
current_char- '  '  )  or  (  current_ehar-'-'  )  loop 
if  current_char- ' - '  then 

exit  when  look_char/~ '  - ' ; 

end  if; 

end  loop; 

end  Ad venae; 
end  LLJTOKENS;  . 


Figure  14-2.  EDSA  Breaks  View  of  Compilation  Unit  LLJTOKENS 
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itMlkMl-ZklM/cck 


Wpln 

prlntvpln  *  *s 

—  Skip  *Mta  ««ct  Md  CMMOtl 

trtill*  <  curr«nt_<twr*MCII.m(  )  or  C  Curran  t_fhar»hscn.  ITT  )  ar  ( 
_  curr»nt_char-'  '  )  «r  (  curran  t_ch*r»-  )  Imp 

■  If  qirrwtjeltpr*'-'  than 

*  |paio*hoad; 

•artt  ahm  look_chor/-'-‘: 

Sfc1pul1n*(  Stmdvdjnput  );  . 
st«rt_*f_1lM  :•  TWK; 
and  If: 

*  Cfcar_Mvmc*; 

If  currant_char-ASCn.t0T  than 
•os  :*  TPUE: 

•1*1f  cvrrMCjcW*'*'  than 

MxCS  trine 

•1*1  f  eurrant_d*»r-"'  than 

altlf  (  cur ran  t_ehir  In  UPK*_C*Sl_lXTTIH  )  or  (  curran t_char  lit 
lOtC^jCMC-lfTm  )  than 
Naat_X«aint1fTar: 

•In 

»mt_Jnt_Syp: 

wxt.orintvalM  :•  tiintnlw: 

•axt.ubldndm  :•  tablalndw: 
naxt. Ilnanuabar  airrnOlM! 
md  hdvwcp; 

•nd  IL.TOXMS: 

BB^^^SOEni  (  far»«r<  I  of  1  ) 

Wmiltlpl.U)  lUttwttl  I 
»naut-«t»taaant 


Figure  144.  EDSA  Screen  of  Statement  Traversal  Using  Control  Flow  In  Unit  LL 
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Figure  14-5.  EDSA  Annotations  Example  In  Compilation  Unit  LL.TOKENS 
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Figure  14-6.  EDSA  Pebbling  Example  in  Compilation  Unit  LL.TOKENS 
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15.  LDRA  Testbed 

The  LDRA  Testbed  provides  both  static  and  dynamic  analysis.  In  static  analysis,  source 
code  is  analyzed  to  give  information  on  control,  data,  and  information  flow,  logical  com¬ 
plexity,  and  procedure  and  variable  usage.  Conformance  to  user-weighted  programming 
standards  is  checked.  Dynamic  analysis  capabilities  provide  assertion  analysis  and  mea¬ 
surement  of  test  completeness  in  terms  of  subcondition,  statement,  branch,  and  Linear  Code 
Sequence  and  Jump  (LCSAJ)  coverage.  Analysis  of  test  data  set  redundancy  is  provided  to 
optimize  the  test  effort  Identification  of  the  test  data  sets  that  execute  each  line  of  code  fa¬ 
cilitates  software  modification. 


15.1  Tool  Overview 

The  LDRA  Testbed  was  developed  by  Liverpool  Data  Research  Associates.  It  is  mar¬ 
keted  by  a  subsidiary  company.  Program  Analysers  Ltd.,  who  also  provide  a  series  of  train¬ 
ing  courses  and  consultancy  services.  Additional  third  party  products  are  available  in 
Europe.  The  Testbed  has  been  commercially  available  since  1974  and  there  are  over  400 
current  licensees.  It  is  available  for  eight  languages  (Ada,  C,  Fortran,  Pascal,  PL/M  86,  PL/ 
1 ,  COBOL,  and  Coral  66)  on  a  wide  range  of  operating  environments.  The  following  partial 
list  exemplifies  the  scope  of  this  range  of  environments:  Apollo  machines  under  Unix,  DEC 
VAX  under  VMS,  Unix,  or  Ultrix,  IBM  under  CMS,  DOS  or  TSO/MVS,  Sun  3  and  Sun  4 
under  Unix,  and  Hewlett-Packard  under  RTEA  or  HPUX.  Using  windowing  capabilities,  a 
graphical  interface  is  available  for  Sun,  Apollo  and,  in  some  cases,  VAX  workstations.  The 
command  line  options  available  for  the  VAX/VMS  and  Unix  environments  differ.  Unix  us¬ 
ers  can  set  options  to  expand  included  files  where  possible,  generate  diagnostic  printouts, 
and  initialize  test  profiles.  Several  additional  options  are  provided  in  the  VAX/VMS  envi¬ 
ronment,  for  example,  the  ability  to  create  a  log  file  of  LDRA  Testbed  usage,  to  limit  the 
type  of  coverage  monitored,  and  to  format  or  pack  the  generated  execution  history. 

The  evaluation  was  performed  on  version  4.8.01  of  the  Ada  testbed  running  on  a  Sun  4 
under  Unix.  At  the  time  of  evaluation,  prices  for  the  testbed  start  at  $12,000,  depending  on 
the  class  of  computer  and  language. 

LDRA  Testbed  is  a  menu-driven  tool.  Its  application  begins  with  static  analysis  of  the 
software  under  test  This  lexical  and  syntactic  analysis  produces  a  reference  listing  contain¬ 
ing  source  code  reformatted  to  LDRA  Testbed  reformatting  standards.  Reference  line  num- 
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bers  are  given  for  each  statement  line.  At  the  same  time,  the  source  code  is  searched  for 
violations  to  the  applicable  set  of  language  standards  provided  with  the  testbed.  These  stan¬ 
dards  check  for  conformance  to  much  of  the  Safe  Ada  Subset.  Reporting  on  any  particular 
standard  is  optional,  the  user  selects  appropriate  standards  by  awarding  penalty  marks 
greater  than  zero  for  violations;  the  static  analysis  produces  a  total  penalty  award  for  the 
analyzed  source  code.  Where  appropriate,  the  user  can  also  specify  acceptable  limits  for 
particular  standards. 

Complexity  analysis  is  based  on  the  control  flow  structure  expressed  in  terms  of  basic 
blocks.  The  complexity  is  reported  in  terms  of  the  number  and  average  length  of  basic 
blocks,  the  number  of  control  flow  knots,  and  cyclomatic  complexity.  In  addition,  two  ap¬ 
proaches  are  used  to  analyze  program  structure.  First,  interval  analysis  reports  on  the  reduc- 
ibility  of  the  software  and  degree  of  nesting.  Second,  the  program  structure  is  evaluated 
against  a  set  of  user-tailorable  language  construct  templates,  an  approach  called  structured 
programming  verification.  Two  further  metrics,  essential  knots  and  McCabe’s  essential 
complexity,  are  provided  to  report  on  unstructuredness.  The  user  specifies  whether  com¬ 
plexity  analysis  should  be  applied  to  all  program  units  or  limited  to  an  identified  set  of  pro¬ 
gram  units.  Kiviat  diagrams  are  provided  for  reporting  of  the  various  complexity  and 
structure  metrics.  These  diagrams  allow  diagramming  multiple  metrics  simultaneously, 
each  with  its  achieved  and  user-defined  upper  and  lower  bounds. 

The  Data  Flow  Analyser  reports  procedure  call  information,  data  flow  anomalies,  and 
procedure  parameter  analysis.  Weak  data  flow  analysis  is  applied  to  identify  undefined  data 
variables  and  defined  variables  that  are  redefined  or  undefined  without  first  referencing  the 
previous  definition.  Procedure  parameter  analysis  classifies  parameters  as  referenced  only, 
defined  only,  both  referenced  and  defined,  or  not  used;  this  analysis  is  carried  out  across 
procedure  boundaries. 

Information  flow  analysis  is  a  new  capability  that  provides  information  on  the  interde¬ 
pendencies  of  program  variables.  LDRA  Testbed  currently  supports  analysis  of  backwards 
strong  and  weak  dependencies  on  a  procedure-by-procedure  basis.  This  capability  can  be 
used  in  two  ways.  First,  as  a  source  of  documentation,  for  example,  to  support  identifying 
the  consequences  of  a  software  change.  Secondly,  the  user  can  specify  information  flow  de¬ 
pendency  assertions  as  special  comments.  The  testbed  then  compares  the  expected  depen¬ 
dencies  with  actual  dependencies,  reporting  the  results. 

The  Cross  Referencer  performs  a  complete  cross-reference  of  all  data  items  used  in  a 
program.  The  type  of  each  data  item  is  classified  as  global,  local,  or  parameter.  For  each 
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procedure,  the  referencer  also  identifies  all  other  procedures  that  this  procedure  calls,  and 
all  procedures  that  call  this  one. 

LGSAJ  analysis  is  the  final  type  of  static  analysis  provided.  It  aids  the  user  in  isolating 
LCSAJs  by  highlighting,  on  a  source  code  listing,  the  start  and  finish  of  the  linear  code  se¬ 
quence  of  each  LCSAJ.  Unreachable  LCSAJs,  and  any  other  unreachable  code  statements, 
are  indicated. 

The  Dynamic  Analyser  instruments  source  code  with  probes  which,  upon  execution, 
write  information  to  an  execution  history  file.  This  is  usually  done  by  writing  to  the  host 
disk  at  run  time.  To  allow  for  host/target  computer  configurations,  however,  the  instrumen¬ 
tation  can  be  adapted  to  channel  the  execution  history  generated  by  the  instrumented  target 
image  back  to  the  host  and  stored  for  subsequent  analysis.  This  may  be  achieved  by  using 
a  spare  serial  line.  Alternatively,  it  may  be  possible  to  arrange  for  storage  of  the  execution 
history  using  an  area  of  memory  on  the  target,  with  this  buffer  subsequently  uploaded  to 
the  host. 

After  instrumentation,  the  user  compiles  and  links  the  instrumented  program  in  the  usu¬ 
al  way.  For  simple  programs,  the  resulting  executable  can  be  run  under  control  of  LDRA 
Testbed,  which  queries  the  user  for  the  names  of  input  and  output  streams.  Alternatively, 
the  program  can  be  executed  independently  of  the  testbed.  In  either  case,  after  the  program 
has  run,  the  user  invokes  the  Dynamic  Coverage  Analyser  to  analyze  the  generated  execu¬ 
tion  history  and  provide  a  name  for  the  current  test  data  set  The  coverage  analyzer  takes 
account  of  the  results  of  previous  test  data  sets  to  accumulate  the  execution  coverage  over 
a  series  of  test  runs.  (The  user  has  no  direct  control  over  adding  an  execution  history  to  the 
accumulated  coverage  data;  this  is  handled  automatically.)  For  each  of  subcondition,  state¬ 
ment,  branch,  and  LCSAJ  coverage,  the  analyzer  provides  a  list  of  the  respective  items  con¬ 
tained  in  the  program  and  identifies  the  old,  new,  and  total  coverage  percentage  achieved 
for  each  item.  Unexecuted  items  are  identified.  In  each  case,  this  is  followed  by  a  summary 
that  reports  the  total  number  of  executable  statements,  branches,  or  LCSAJs,  as  appropri¬ 
ate,  the  number  that  were  executed,  the  number  not  executed,  and  the  corresponding  test 
effectiveness  metric. 

The  user  may  request  a  dynamic  trace  to  explicitly  show  the  flow  of  control  resulting 
from  the  test  data  set  This  trace  may  be  limited  to  specified  procedures,  or  to  between  a 
user-specified  range  of  code  line  numbers.  The  LDRA  Testbed  will  override  this  request  if 
the  resulting  display  will  be  too  large. 
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The  testbed  uses  three  Test  Effectiveness  Ratio  (TER)  metrics  to  report  on  the  effec¬ 
tiveness  of  the  test  data: 

•  TER1  =  Number  of  statements  exercised  at  least  once  /  Total  number  of  statements 

•  TER2  =  Number  of  branches  exercised  at  least  once  /  Total  number  of  branches 

•  TER3  =  Number  of  LCSAJs  exercised  at  least  once  /  Total  number  of  LCSAJs 

In  terms  of  coverage,  TER3  lies  between  branch  and  path  coverage.  That  is,  LCSAJs  pro¬ 
vide  a  measure  that  is  more  stringent  than  branch  coverage  without  incurring  the  overhead 
of  path  coverage.  Additionally,  LDRA  Testbed  reports  the  number  of  overlapping  LCSAJs 
containing  each  reformatted  statement  as  the  “density.”  This  figure  can  be  used  as  a  mea¬ 
sure  of  the  complexity  encountered  when  reading  or  modifying  the  program. 

When  a  program  contains  tasks,  the  generated  execution  history  will  contain  the  inter¬ 
leaved  execution  histories  of  those  tasks.  LDRA  Testbed  can  distinguish  between  these 
multiple  histories,  but  some  special  user  actions  are  required  to  assist  in  the  processing  of 
the  separated  histories. 

The  user  can  embed  assertions  in  Ada  comments.  These  special  comments  can  be  used 
to  specify  pre-  or  post-conditions  applying  to  a  section  of  code,  check  that  inputs  satisfy 
predetermined  ranges,  or  check  that  loop  and  array  indices  are  within  bounds.  When  con¬ 
formance  checking  is  switched  on,  the  testbed  translates  the  special  comments  to  execut¬ 
able  code  and  inserts  a  user-tailorable  failure  handling  routine.  The  supplied  failure 
handling  routine  simply  prints  a  message  identifying  the  failing  assertion  and  then  raises  a 
fail  exception.  It  is  the  user’s  responsibility  to  determine  appropriate  assertion  conditions 
and  to  ensure  that  the  assertions  are  positioned  where  valid  executable  statements  are  al¬ 
lowed  in  the  source  code.  The  assertion  format  and,  to  some  extent,  syntax  and  semantics 
are  tailorable  via  means  of  a  parameter  file. 

Two  final  capabilities  provide  some  limited  support  for  regression  analysis.  A  Profile 
Analyzer  is  provided  to  compare  the  coverage  profiles  generated  by  a  series  of  test  data 
sets.  It  identifies  any  data  set(s)  that  are  redundant,  that  is,  those  that  do  not  contribute  to 
increasing  the  overall  coverage.  Where  two  or  more  redundant  test  data  sets  are  identified, 
LDRA  Testbed  will  recommend  removal  of  the  one  that  generates  the  largest  execution  his¬ 
tory.  For  each  executable  line  of  code,  the  Dynamic  Data  Set  Analysis  option  identifies  the 
test  data  set(s)  that  execute  that  line.  This  allows  the  user  to  determine  which  test  data  sets 
are  affected  by  a  modification  and,  therefore,  the  tests  that  must  be  repeated. 
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The  results  of  testbed  analysis  are  examined  using  a  viewing  option.  They  can  be 
viewed  at  either  the  compilation  unit  level  or  system  level  (that  is,  for  the  full  set  of  com¬ 
pilation  units).  Various  textual  displays  are  available  or  the  user  may  access  a  submenu  of 
graphical  displays.  Navigation  through  textual  displays  is  command  driven.  Graphical  dis¬ 
plays  are  available  as  bar  charts  of  complexity  and  coverage  measures,  Kiviat  diagrams  of 
quality  metrics,  flowgraphs  of  the  software  control  flow  graph,  and  call-trees  showing  the 
procedure  hierarchy.  Static  and  dynamic  views  of  both  call-trees  and  flowgraphs  are  avail¬ 
able.  The  static  control  flow  graph  can  be  annotated  with  the  results  of  coverage  analysis. 
In  addition,  the  user  can  request  an  active  flowgraph  that  illustrates  the  execution  achieved 
by  the  last  test  data  set.  Navigation  through  the  graphical  displays  is  provided  via  selection 
from  a  set  of  icons  that  support  such  functions  as  automatic  zooming  and  printing  a  screen. 


15.2  Observations 

Ease  of  use.  Overall,  LDRA  Testbed  is  very  easy  to  use  and  provides  a  broad  range  of 
testing  facilities.  It  automates  all  repetitive  tasks  and  requires  no  redundant  user  input;  for 
example,  a  special  script  is  provided  to  facilitate  testing  of  software  composed  of  many 
source  modules  in  separate  files.  This  script,  called  tbset ,  allows  a  user  to  associate  a  name 
with  a  set  of  files  and  manipulate,  list,  and  select  sets.  LDRA  Testbed  can  be  invoked  from 
tbset  to  apply  user-selected  testbed  operations  to  a  chosen  set  of  files  as  a  group.  In  this 
mode,  however,  some  usual  testbed  options  are  not  available;  in  particular,  the  user  cannot 
limit  processing  to  a  named  set  of  files  or  limit  reporting  to  a  named  set  of  program  units. 
In  addition,  only  system-wide  analysis  results,  a  call-tree  display,  and  a  variety  of  flow 
graphs  are  available  for  viewing.  (To  access  results  for  particular  software  units,  the  user 
can  view  results  through  the  testbed  directly.)  If  necessary,  tbset  allows  the  user  to  spawn 
a  shell  script  for  non-testbed  related  processing. 

The  Management  Summary  report  provides  useful  high-level  information  on  the  qual¬ 
ity  of  the  software  and  on  the  effectiveness  of  testing  to  date.  More  detailed  information  is 
provided  in  a  series  of  analysis  reports,  some  of  which  are  very  lengthy.  While  some  users 
may  find  the  provision  of  multiple  alternative  complexity  measures  useful,  others  will  find 
much  of  this  information  redundant 

Graphical  outputs  are  available  on  Sun,  Apollo,  HP,  IBM  RS6000,  VAX,  and  most  oth¬ 
er  types  of  workstations  with  windowing  capabilities,  histograms  drawn  in  orthographic 
projection  are  available  for  any  terminal  supporting  VT100  graphics.  These  histograms  are 
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used  to  profile  coverage  information  and  summarize  information  on  the  program  quality, 
complexity  and  structuredness.  Full  color  is  available  for  graphical  displays.  All  graphics 
displays  can  be  exported  as  Postscript  files. 

Documentation  and  user  support.  The  supplied  documentation  is  well-written  and 
comprehensive.  It  includes  a  standard  interface  file  to  facilitate  using  LDRA  Testbed  out¬ 
puts  as  inputs  to  other  tools.  This  file  allows  testbed  information  to  be  viewed  at  three  lev¬ 
els:  procedure/function,  source  module,  and  project  (that  is,  some  related  set  of  source 
modules).  Through  this  interfacing  facility,  LDRA  Testbed  has  been  used  with  StP,  Team¬ 
work,  Typhoon,  System  Engineer,  Mascot,  Infomix,  and  ASA  CASE  tools,  and  with  the 
TBGEN  testing  tool. 

In  all  instances,  the  staff  at  Program  Analysers  were  helpful  and  friendly  and  provided 
quick  resolution  of  encountered  problems. 

Instrumentation  overhead.  Full  instrumentation  of  the  Ada  Lexical  Generator  for 
statement,  branch,  and  LCSAJ  coverage  gave  a  source  code  increase  of  nearly  100%.  The 
size  of  the  instrumented  executable  program  increased  approximately  12%.  The  user  can 
limit  the  amount  of  instrumentation  performed  by  requesting  monitoring  of  only  statement 
coverage,  or  only  statement  and  branch  coverage.  Since  the  user  specifies  the  files  which 
are  to  be  instrumented,  instrumentation  can  be  restricted  to  specific  compilation  units.  It 
cannot,  however,  be  limited  to  specific  program  units  within  a  compilation  unit 

Ada  restrictions.  The  LDRA  Testbed  supports  full  Ada.  However,  the  documentation 
lists  the  following  constraints  for  version  4.8.01  of  the  Testbed: 

•  For  static  analysis  and  cross-referencing:  ( 1 )  The  use  of  generics  may  not  be  correctly 
handled  in  some  cases,  (2)  Analyses  are  limited  to  variables  in  the  current  module,  (3) 
Overloaded  procedures  may  cause  misleading  messages  about  recursion,  and  (4) 
Some  combinations  of  literal  procedure  parameters  are  incorrectly  analyzed. 

•  Calls  between  procedures  in  package  bodies  are  only  handled  if  their  declaratives 
appear  textually  before  use. 

•  Incorrect  branches  are  generated  for  certain  nested  select  statements. 

•  In  the  case  of  information  flow  analysis,  strongly-defined  variables  may  be  miscate- 
gorized  in  the  presence  of  exception  handlers. 

•  The  analysis  may  be  incorrect  for  loops  implementing  recursive  functions  of  degree 
greater  than  two. 

Problems  encountered.  Difficulties  encountered  in  installing  an  earlier  version  of  the 
testbed  have  been  resolved.  LDRA  Testbed  performed  as  described  in  the  documentation. 
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15.3  Planned  Additions 

Currently  under  beta  testing,  dynamic  data  flow  testing  for  Ada  is  expected  to  become 
available  in  autumn  1992.  Also  under  development  are  system- wide  data  flow  analysis  and 
the  assessment  and  reporting  of  reliability  metrics. 

15.4  Sample  Outputs 

Figures  15-1  through  15-19  provide  sample  outputs  from  LDRA  Testbed. 


15-7 


LDRA  Testbed 


PART  II 


MANAGEMENT  SUMMARY 


TESTBED  VERSION  :  4.8.01 

FILE  UNDER  TEST  :  adaler_dir_2/ll_coapile. a  9 

DATE  OF  ANALYSIS  :  Mon  Oot  1211: 53: 18  EDT  1992 

STANDARDS  VIOLATIONS  IN  STATIC  ANALYSIS 


LINE 

NUMBER  VIOLATION 


PENALTY 

MARX 


28 

1-0  package 

1 

82 

USE  clause 

1 

82 

1-0  package 

1 

85 

Exception  declaration 

1 

88 

Number  Declaration 

1 

95 

Predefined  language  environment  naae 

"INTEGER" 

1 

688 

Identical  naae  in  scope  * TEST SYNCH" 

1 

694 

Predefined  language  environment  name 

■INTEGER’ 

1 

695 

Identical  name  in  another  scope  *1* 

1 

695 

Predefined  language  environment  name 

■INTEGER* 

1 

744 

Predefined  language  environment  name 

•FALSE* 

1 

752 

Raise  statement 

1 

782 

Predefined  language  environment  name 

■TRUE' 

1 

797 

Predefined  language  environment  name 

•TRUE* 

1 

TOTAL  PENALTY  FROM  STATIC  ANALYSIS  -  125 
TOTAL  NUMBER  OF  LINES  IN  PROGRAM  -  868 


SUMMARY  OF  EXECUTABLE  BODIES 


NAME 

START 

LINE 

NO  OF 
LINES 

LLFIND 

152 

22 

LLPRT STRING 

179. 

10 

LLPRTTOKEN 

194 

11 

LLSKIPTOKEN 

210 

10 

LL5KIPN0DE 

225 

12 

LLSKIPBOTH 

244 

13 

LLFATAL 

262 

9 

GET_CHARACTER 

278 

14 

CVT_STRING 

307 

10 

MAXE_TOKEN 

318 

66 

Figure  15-1.  LDRA  Testbed  Management  Summary  for  LL.COMPILE 
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LLNEXTTOKEN 

400 

8 

BUILDRIGHT 

455 

66 

BUILDSELECT 

557 

8 

READGRAM 

537 

39 

ERASE 

587 

14 

MATCH 

630 

17 

EXPAND 

639 

47 

SYNCHRONIZE 

697 

63 

TEST  SYNCH 

761 

15 

PARSE 

778 

75 

LINAIN 

655 

7 

LL  COMPILE 

864 

5 

THERE  ARE  1  UNREACHABLE  LCSAJS 

THE  MAXIMUM  LCSAJ  DENSITY  IS  IS  AT  LINE  456 

THERE  AXE  1  SEQUENCES  OF  UNREACHABLE  CODE 

THE  LONGEST  IS  3  LINES  AT  LINE  167 

THE  TOTAL  NUMBER  OF  UNREACHABLE  LINES  IS  10 

THERE  ARE  7  UNREACHABLE  BRANCHES 

1COMPLEXITY  ANALYSIS  PRODUCES  THE  FOLLOWING  TABLE  OF  RESULTS 


PROCEDURE 

EXEC. 

LINES 

BASIC 

BLOCKS 

AVG. 

LEN. 

ORDER  1 
INTER  V. 

MAX  ORDER 
INTERV.  REDUC. 

MCCABE 

KNOTS 

MCCABE 

E 

KNOTS 

LL.COMPILE 

37 

1 

37.00 

1 

1 

YES 

1 

0 

1 

0 

LLFIND 

31 

13 

1.63 

3 

3 

YES 

5 

14 

4 

6 

LLPRTSTRING 

10 

5 

3.00 

2 

2 

YES 

3 

3 

3 

2 

LLPRTTOKEN 

10 

4 

2.50 

1 

1 

YES 

2 

1 

1 

0 

LLSKIPTOKEN 

10 

1 

10.00 

1 

1 

YES 

1 

0 

1 

0 

LL5XIPN0DE 

13 

1 

13.00 

1 

1 

YES 

1 

0 

1 

0 

LLSXIPBOTH 

13 

1 

13.00 

1 

.  1 

YES 

1 

0 

1 

0 

LLFATAL 

9 

3 

4.50 

1 

1 

YES 

1 

0 

1 

0 

GET_CHARACTER 

14 

6 

3.33 

1 

1 

YES 

3 

2 

1 

0 

MAXITOKEN 

68 

17 

4.00 

1 

1 

YES 

11 

30 

1 

0 

LLHKXTTOKEN 

8 

3 

2.67 

1 

1 

YES 

2 

0 

1 

0 

LIMA  IN 

14 

1 

14.00 

1 

1 

YES 

1 

0 

1 

0 

CVT_STRING 

10 

7 

1.43 

2 

2 

YES 

3 

2 

1 

0 

READGRAM 

38 

14 

2.71 

5 

3 

YES 

6 

5 

1 

0 

PARSE 

73 

33 

3.17 

2 

3 

YES 

11 

11 

1 

0 

BUILDRIGHT 

63 

30 

3.10 

2 

3 

YES 

10 

30 

8 

26 

BUTT, PS  ELECT 

8 

4 

3.00 

2 

3 

YES 

2 

1 

1 

0 

ERASE 

11 

6 

1.83 

2 

3 

YES 

3 

4 

3 

4 

EXP AMD 

43 

15 

3.87 

4 

3 

YES 

7 

3 

1 

0 

TEST SYNC* 

14 

7 

2.00 

3 

3 

YES 

3 

2 

1 

0 

Figure  15-1  continued:  LORA  Testbed  Management  Summary  for  LL_COMPILE 
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MATCH 

16 

9 

1.78 

2 

2 

YES 

4 

9 

SYNCHRONIZE 

59 

23 

2.57 

7 

3 

YES 

9 

11 

TOTAL 

550 

183 

3.01 

23 

3 

YES 

69 

116 

THE  PROGRAM 

CONTAINS 

22  1 

PROCEDURES 

1 STANDARDS  VIOLATIONS  IN  COMPLEXITY  ANALYSIS 


PROCEDURE 


VIOLATION 


PENALTY 


LLFIND  CONTAINS  ESSENTIAL  KNOTS 

LLPRTSTRING  CONTAINS  ESSENTIAL  KNOTS 

MAKE  TOKEN  MCCABE  MEASURE  GREATER  THAN 

PARSE  MCCABE  MEASURE  GREATER  THAN 

BUILDRIGHT  CONTAINS  ESSENTIAL  KNOTS 

ERASE  CONTAINS  ESSENTIAL  KNOTS 

MATCH  CONTAINS  ESSENTIAL  KNOTS 

SYNCHRONIZE  CONTAINS  ESSENTIAL  KNOTS 

TOTAL  PENALTY  FROM  COMPLEXITY  ANALYSIS  -  8 


10 

10 


1 

1 

1 

1 

1 

1 

1 

1 


1DATA  FLOW  ANALYSIS  RESULTS 


1  VARIABLES  MERE  DECLARED  BUT  NEVER  USED 

40  TYPE  UR  ANOMALIES  FOUND 
17  TYPE  DU  ANOMALIES  FOUND 

41  TYPE  DD  ANOMALIES  FOUND 


1DTNAMIC  COVERAGE  ANALYSIS  REPORT 


PROFILES  INCLUDED  FOR  THE  FOLLOWING  TEST  DATA  SETS 


1)  tutl.lu 
3)  sample. lex 

DYNAMIC  ANALYSIS  WARNINGS 


8  MISSING  LINEAR  CODE  SEQUENCE  AND  JUMP  TRIPLES 
6  MISSING  BRANCHES 

1STATEKENT  EXECUTION  HISTORY  SUMMARY 


EXECUTABLE 

STATBOENTS 

NUMBER  EXECUTED 

OLD  NEW  TOTAL 

OLD 

TER  1 
NEW 

TOTAL 

LL_COMPILE 

27 

27 

27 

27 

1.00 

1.00 

1.00 

LLFIND 

17 

16 

17 

17 

0.94 

1.00 

1.00 

LLPRTSTRING 

10 

0 

0 

0 

0.00 

0.00 

0.00 

LLPRTTOXEN 

10 

0 

0 

0 

0.00 

0.00 

0.00 

LLSKIPTOKEN 

10 

0 

0 

0 

0.00 

0.00 

0.00 

Figure  15-1  continued:  LDRA  Testbed  Management  Summary  for  LL_COMPILE 
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> 


> 


> 


> 


> 


I 


> 


> 


LLSKIPNODE 

12 

0 

0 

0 

0.00 

0.00 

0.00 

LLSKIPBOTH 

13 

0 

0 

0 

0.00 

0.00 

0.00 

LLFATAL 

e 

0 

0 

0 

0.00 

0.00 

0.00 

GET_CHARACTER 

14 

0 

0 

0 

0.00 

0.00 

0.00 

MAKETOICEN 

67 

0 

0 

0 

0.00 

0.  00 

0.00 

LLNEXTTOKEN 

8 

8 

8 

8 

1.00 

1.00 

1.00 

LLMAIN 

14 

14 

14 

14 

1.00 

1.00 

1.00 

CVT_STRING 

9 

0 

0 

0 

0.00 

0.00 

0.00 

READGRAM 

36 

38 

38 

38 

1.00 

1.00 

1.00 

PARSE 

73 

56 

56 

56 

0.77 

0.77 

0.77 

BUILDRIGHT 

62 

54 

.54 

54 

0.87 

0.87 

0.87 

BUILD  SELECT 

8 

8 

8 

8 

1.00 

1.00 

1.00 

ERASE 

11 

11 

11 

11 

1.00 

1.00 

1.00 

EXPAND 

43 

38 

38 

38 

0.88 

0.88 

0.88 

TESTSTNCH 

14 

0 

0 

0 

0.00 

0.00 

0.00 

MATCH 

15 

13 

13 

13 

0.87 

0.87 

0.87 

SYNCHRONIZE 

57 

0 

0 

0 

0.00 

0.00 

0.00 

TOTAL 

540 

283 

284 

284 

0.52 

0.53 

0.53 

SUB-CONDITIONS 

SUMMARY 

SUB-CONDITIONS 

NUMBER  EXECUTED 

OLD  NEW  TOTAL 

OLD 

TER  CON 

NEW 

TOTAL 

LL_COMPILE 

PROCEDURE 

II 

NO 

SUB-CONDITIONS 

LLPIND 

PROCEDURE  CONTAINS 

NO 

SUB-CONDITIONS 

LLPRT STRING 

PROCEDURE 

CONTAINS 

NO 

SUB-CONDITIONS 

LLPRTTOKEN 

PROCEDURE 

CONTAINS 

NO 

SUB-CONDITIONS 

LLSKIPTOKEN 

PROCEDURE 

CONTAINS 

NO 

SUB— CONDITIONS 

LLSKIPNODE 

PROCEDURE 

CONTAINS 

NO 

SUB^CONDITIONS 

LLSKIPBOTH 

PROCEDURE 

CONTAINS  NO 

SUB-CONDITIONS 

LLFATAL 

PROCEDURE 

CONTAINS 

NO 

SUB-CONDITIONS 

GET_CHARACTER 

PROCEDURE 

CONTAINS 

NO 

SUB-CONDITIONS 

MAXX_TOKEN 

PROCEDURE 

CONTAINS 

NO 

SUB-CONDITIONS 

BUXLDRIGHT 

PROCEDURE 

CONTAINS 

NO 

SUB-CONDITIONS 

BUILD SELECT 

PROCEDURE 

CONTAINS 

NO 

SUB-CONDITIONS 

ERASE 

PROCEDURE  CONTAINS 

NO 

SUB-CONDITIONS 

EXPAND 

8 

8 

8  8 

1.00 

1.00 

1.00 

TESTSTNCH 

PROCEDURE 

CONTAINS 

NO 

SUB-CONDITIONS 

MATCH 

4 

4 

4  4 

1.00 

1.00 

1.00 

STNCHRONIEE 

12 

0 

0  0 

0.00 

0.00 

0.00 

TOTAL 

24 

12 

12  12 

0.50 

0.50 

0.50 

Figure  15-1  continued:  LDRA  Testbed  Management  Summary  for  LL_COMPILE 
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1BKANCH  EXECUTION  HISTORY  SUMMARY 


BRANCHES 

NUMBER  EXECUTED 

OLD  NEW  TOTAL 

OLD 

TER  2 

NEW 

TOTAL 

LL  COMPILE 

13 

13 

13 

13 

1.00 

1.00 

1.00 

LLFIND 

34 

11 

13 

13 

0.33 

0.35 

0.35 

LLPRTSTRING 

9 

0 

0 

0 

0.00 

0.00 

0.00 

LLPRTTOKEN 

9 

0 

0 

0 

0.00 

0.00 

0.00 

LLSKIPTOKEN 

3 

0 

0 

0 

0.00 

0.00 

0.00 

LLSKIPNODE 

3 

0 

0 

0 

0.00 

0.00 

0.00 

LLSKIPBOTH 

3 

0 

0 

0 

0.00 

0.00 

0.00 

LLFATAL 

3 

0 

0 

0 

0.00 

0.00 

0.00 

GET  CHARACTER 

7 

0 

0 

0 

0.00 

0.00 

0.00 

MAKE_TOKEN 

31 

0 

0 

0 

0.00 

0.00 

0.00 

LLNEXTTOKEN 

5 

4 

4 

4 

0.80 

0.80 

0.80 

LLMAIN 

5 

S 

5 

5 

1.00 

1.00 

1.00 

CVT_STRING 

7 

0 

0 

0 

0.00 

0.00 

0.00 

REAOGRAM 

30 

30 

30 

30 

1.00 

1.00 

1.00 

PARSE 

39 

35 

35 

35 

0.64 

0.64 

0.64 

BUILDRIGHT 

26 

33 

33 

33 

0.85 

0.85 

0.85 

BUILDSELECT 

4 

4 

4 

4 

1.00 

1.00 

1.00 

ERASE 

7 

7 

7 

7 

1.00 

1.00 

1.00 

EXPAND 

IB 

15 

15 

15 

0.83 

0.83 

0.83 

TEST  SYNCH 

11 

0 

0 

0 

0.00 

0.00 

0.00 

MATCH 

11 

7 

7 

7 

0.64 

0.64 

0.64 

SYNCHRONISE 

36 

0 

0 

0 

0.00 

0.00 

0.00 

TOTAL 

393 

133 

134 

134 

0.46 

0.46 

0.46 

1LCSAJ  EXECUTION  HISTORY  SUMfARY 


LCSAJS 

NUMBER  EXECUTED 

OLD  NEW  TOTAL 

OLD 

TER  3 
NEW 

TOTAL 

LL_CONPILE 

13 

13 

13 

13 

1.00 

1.00 

1.00 

LLTIND 

34 

10 

11 

11 

0.29 

0.33 

0.32 

LLPRTSTRING 

10 

0 

0 

0 

0.00 

0.00 

0.00 

LLPRTTOKEN 

13 

0 

0 

0 

0.00 

0.00 

0.00 

LLSKIPTOKEN 

3 

0 

0 

0 

0.00 

0.00 

0.00 

Figure  15-1  continued:  LDRA  Testbed  Management  Summary  for  LL_COMPILE 
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LLSKIPNODE 

3 

0 

0 

0 

0.00 

0.00 

0.00 

LLSKIPBOTH 

3 

0 

0 

0 

0.00 

0.00 

0.00 

LLFATAL 

a 

0 

0 

0 

0.00 

0.00 

0. 00 

GET_CHARACTER 

6 

0 

0 

0 

0.00 

0.00 

0.00 

MAKE_30KEN 

33 

0 

0 

0 

0.00 

0.00 

0.00 

LLNEXTTOKEN 

7 

3 

3 

3 

0.43 

0.43 

0.43 

LUiAXN 

S 

5 

5 

5 

1.00 

1.00 

1.00 

CVT_STRING 

9 

0 

0 

0 

0.00 

0.00 

0.00 

READCRAM 

25 

20 

20 

20 

0 . 80 

0.80 

0.80 

PARSE 

34 

23 

23 

23 

0.68 

0.68 

0.68 

BUILDRIGHT 

32 

24 

24 

24 

0.75 

0.75 

0.75 

BUILDSELECT 

5 

4 

4 

4 

0.80 

0.80 

0.80 

ERASE 

8 

7 

7 

7 

0.88 

0.88 

0.88 

EXPAND 

20 

15 

15 

15 

0.75 

0.75 

0.75 

TESTSYNCH 

12 

0 

0 

0 

0.00 

0.00 

0.00 

MATCH 

12 

6 

6 

6 

0.50 

0.50 

0.50 

SYNCHRONIZE 

38 

0 

0 

0 

0.00 

0.00 

0.00 

TOTAL 

326 

130 

131 

131 

0.40 

0.40 

0.40 

isummary  or  efeect  or  current  test  data  set  on  the  coverage  metrics 


PROCEDURE  NAME 

TER  1 

TER  2 

TER  3 

LL_C0HPILE 

1.00 

1.00 

1.00 

LLPIND 

lnex*aa*d 

Incraaaad 

inarmaamd 

LLPRTSTR3NG 

No  Chang* 

No  Chang* 

NO  Chang* 

LLPRTTOKEN 

No  Chujt 

No  Chang* 

No  Chang* 

LLSKIPTOKEN 

NO  Chang* 

No  Chang* 

NO  Chang* 

LLSKIPNODE 

No  Chang* 

No  Chang* 

No  Chang* 

LLSKXPBOTH 

No  Chang* 

No  Chang* 

No  Chang* 

LLFATAL 

No  Chang* 

No  Chang* 

No  Chang* 

GET_CHARACTER 

No  Chang* 

No  Chang* 

No  Chang* 

MAXEjrOKEN 

No  Chaag* 

No  Chang* 

No  Chang* 

LLNEXTTOKEN 

1.00 

No  Chang* 

No  Chang* 

LLMAIN 

1.00 

1.00 

1.00 

CVT_STRING 

No  Chang* 

No  Chang* 

No  Chang* 

READCRAM 

1.00 

1.00 

No  Chang* 

PARSE 

No  Chang* 

No  Chang* 

No  Chang* 

BUILDRIGHT 

No  Chang* 

No  Chang* 

No  Chang* 

BUILDSELECT 

1.00 

1.00 

No  Chang* 

ERASE 

1.00 

1.00 

No  Chang* 

EXPAND 

No  Chang* 

No  Chang* 

No  Chang* 

TESTSTNCH 

No  Chang* 

No  Chang* 

No  Chang* 

MATCH 

No  Chang* 

No  Chang* 

No  Chang* 

SYNCHRONIZE 

No  Chang* 

No  Chang* 

No  Chang* 

Figure  15-1  continued:  LORA  Testbed  Management  Summary  for  LL_COMP!LE 
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Figure  15-2.  LDRA  Testbed  Static  Call  Tree  of  LL.COMPILE 
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PROCEDURE  CALL  INFORMATION 


THE  MAIN  PROGRAM 

BETWEEN  LINES  31  AND  868 

LL_CONPILE 

CALLS  THE  FOLLOWING  PROCEDURES 
LLHAIN 


PROCEDURE 

LLFIND 

BETWEEN  LINES  146  AND  174 

DOES  NOT  CALL  ANT  INTERNAL  PROCEDURES 

IS  CALLED  BT  THE  FOLLOWING  PROCEDURES 

MAKE_TOKEN 

PARSE 


PROCEDURE 

LLPRTSTRING 

BETWEEN  LINES  176  AMD  IBB 

DOES  NOT  CALL  ANT  INTERNAL  PROCEDURES 

IS  CALLED  BT  THE  FOLLOWING  PROCEDURES 

LLPRTTOKBl 

LLSKIFNODE 

LLSKIPBOTH 

STNCHRONIEE 


1— - —..I.—. — 

THE  FOLLOWING  VARIABLES  WERE  DECLARED  BUT  NEVER  USED 
VARIABLE  DECLARED  ON  LINE 


TABLE INDEX 


453 


Figure  15-4.  LORA  Testbed  Data  Flow  Analysis  of  LL_COMPILE 
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VARIABLE 


TYPE  UK  ANOMALIES 
UNDEPINE 


REFERENCE 


' GLOBAL ' STAND ARD_ERROR 
180 

'  GLOBAL '  STAND ARD_ERBOR 
300 

RHSARRAY  436 

' GLOBAL' LLTABLESIEE 
136 

LLSYHBOLTABLE 

136 

LLEOTOKS  132 

LLCURTOX . PRINTVALUE 
135 

LLCURTOK . TABLEINDEX 
135 

LLCURTOK . LINENUMBER 
135 

LLCURTOK . ATTRIBUTE 

135 


180 

200 

859 

IN 

PROCEDURE 

PARSE 

136 

866 

IN 

PROCEDURE 

LLMAIN 

666 

IN 

PROCEDURE 

LLMAIN 

866 

IN 

PROCEDURE 

LLMAIN 

866 

IN 

PROCEDURE 

LLMAIN 

866 

IN 

PROCEDURE 

LLMAIN 

866 

IN 

PROCEDURE 

LLMAIN 

1- 


VARIABLE 


TYPE  DU  ANOMALIES 
DEFINE 


UNDEFINE 


CHIUJCOUNT 

457 

520 

I 

529 

533 

LOCOFANY 

788 

852 

PRODUCTIONS 

857 

IN 

PROCEDURE 

READGRAM 

861 

RHSARRAY 

857 

IN 

PROCEDURE 

READGRAM 

861 

THISKHS 

857 

IN 

PROCEDURE 

READGRAM 

861 

LLSTACK . DATA 

866 

IN 

PROCEDURE 

LLMAIN 

868 

LLSTACK . ATTRIBUTE 

866 

IN 

PROCEDURE 

LLMAIN 

868 

LLSTACK.  PARENT 

866 

IN 

PROCEDURE 

LLMAIN 

868 

LLSTACK. TOP 

866 

IN 

PROCEDURE 

LLMAIN 

868 

LLSTACK . LASTCRILD 

866 

IN 

PROCEDURE 

LLMAIN 

868 

LLSYHBOLTABLE 

866 

IN 

PROCEDURE 

LLMAIN 

868 

LLCURTOK . TABLEINDEX 

866 

IN 

PROCEDURE 

LU4AIN 

868 

LLCURTOK . PRINTYALUE 

866 

IN 

PROCEDURE 

LLMAIN 

868 

LL8ENIPTR 

866 

IN 

PROCEDURE 

LLNAIN 

868 

LLLOCEOS 

866 

IN 

PROCEDURE 

LLMAIN 

868 

LLAD VANCE 

866 

IN 

PROCEDURE 

LLNAIN 

868 

Figure  154  continued:  LDRA  Testbed  Data  Flow  Analysis  of  LL_COMPILE 
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VARIABLE 


TYPE  DD  ANOMALIES 
DEFINE 


REDEFINE 


LLTOP  598  594 

LL  STACK . LASTCHILD 

669  671 

LL  STACK . TOP  669  671 


LLSTACX . ATTRIBUTE 
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Figure  15*5.  LDRA  Testbed  Information  Flow  Analysis  for  LLFIND 
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Figure  15-6.  LORA  Testbed  Complexity  Analysis  for  LLFIND 
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Figure  15-6  continued:  LDRA  Testbed  Complexity  Analysis  for  LLFIND 
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Figure  15-6  continued:  LDRA  Testbed  Complexity  Analysis  for  LLFIND 
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Figure  15-6  continued:  LDRA  Testbed  Complexity  Analysis  for  LLFIND 
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Figure15-6  continued:  LDRA  Testbed  Complexity  Analysis  for  LLFIND 
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Figure  15*9.  LDRA  Testbed  Klvlat  Graph  for  LLFIND 
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Figure  15-10.  LDRA  Testbed  LCSAJ  Analysis  for  LL.COMPILE 
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Figure  15-10  continued:  LORA  Testbed  LCSAJ  Analysis  for  LL.COMPILE 
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Figure  15-11.  LDRA  Testbed  Cross  Reference  Analysis  for  LLFIND 
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Figure  15-12.  LDRA  Testbed  Dynamic  Analysis  for  LL_COMPILE 
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Figure  15-12  continued:  LDRA  Testbed  Dynamic  Analysis  for  LL_COMPILE 
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Figure  15-15.  LDRA  Testbed  System  View  Test  Path  (LCSAJ)  Coverage 
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Figure  15*16.  LDRA  Testbed  Coverage  Achieved  Comparison 
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Figure  15-17.  LORA  Testbed  Active  Flowgraph  of  READGRAM 
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Figure  15-18.  LDRA  Testbed  Data  Set  Analysis  for  LLFIND 
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Figure  15-19.  LDRA  Testbed  Profile  Analysis 


15-45 


PART  II 


Logiscope 


I* 


16.  Logiscope 

Logiscope  employs  the  RADC  quality  metrics  model  to  provide  analysis  of  a  set  of 
user-tailorable  quality  metrics  at  both  the  unit  and  integration  levels.  It  provides  coverage 
analysis  of  statement  blocks,  branches,  and  LCSAJs  at  the  unit  level,  and  procedure- to-pro- 
cedure  path  coverage  analysis  at  the  integration  level.  Additional  capabilities  include  the 
generation  of  control  and  call  graphs,  structure  analysis,  and  pseudo-code  generation  to 
support  re-engineering. 

Logiscope  is  one  element  of  a  comprehensive  suite  of  CASE  tools.  AGE/ASA  is  a 
CASE  tool  supporting  functional  specification  activities.  Based  on  IDEFO  and  finite  state 
machine  specification  methods,  it  supports  simulation  and  various  static  analyses  including 
complexity  analysis.  It  also  provides  test  scenario  generation  for  automatic  production  of 
functional  test  suites  which  can  be  fed  into  the  simulator  or  used  during  code  acceptance 
testing  to  ensure  compliance  with  requirements.  Scenario  coverage  can  be  measured  during 
simulation.  Support  for  design  is  available  through  AGE/GEODE.  This  tool  is  based  on  the 
Consulting  Committee  on  International  Telegraph  and  Telephone  (CCITT)  standardized 
language  Specification  and  Description  Language  (SDL)  and  provides  for  design  and  sim¬ 
ulation  of  real-time  software  with  automatic  code  and  application  generation.  AGE/GE¬ 
ODE  also  provides  test  process  generation  to  allow  independently  testing  the  coherence  of 
a  process  with  respect  to  the  rest  of  the  design  prior  to  system  integration.  A  new  tool, 
VEDA,  supports  simulation  and  validation  of  protocols  specified  in  the  International  Orga¬ 
nization  for  Standardization  (ISO)  standard  language  Estelle.  Finally,  DocBuilder  is  used 
to  produce  software  documentation  that  can  be  configured  to  such  standards  as  DoD- STD- 
21 67A.  It  is  based  on  the  Standard  Generalized  Markup  Language  (SGML)  ISO  Standard 
8879. 

16.1  Tool  Overview 

Logiscope  was  developed  by  Verilog,  a  European  company  formed  in  1984.  It  has  been 
available  since  1985  and  there  are  over  5,000  users  worldwide.  Logiscope  is  marketed  in 
the  U.S.  by  Verilog,  Inc.,  the  U.S.  subsidiary.  This  company  also  provides  consulting  and 
training  services,  and  hot-line  support  for  tool  users. 

Logiscope  is  available  for  over  eighty  languages  and  dialects,  including  Ada,  C,  C++, 
and  Fortran.  It  is  supported  on  a  variety  of  workstations  and  mainframes  under  both  Unix 
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and  VMS,  with  graphic  capabilities  available  through  a  number  of  windowing  systems.  As 
with  all  its  tools,  Verilog  has  focused  on  compatibility  of  Logiscope  with  international  stan¬ 
dards  such  as  the  Portable  Common  Tool  Environment  (PcTE),  SDL,  etc.  Logiscope  can 
be  integrated  with  DecFuse,  HP’s  SoftBench,  and  Software  Back  Plane.  It  supports  host/ 
target  testing  via  use  of  a  serial  port  between  the  host  and  target  machines. 

The  evaluation  was  performed  on  Logiscope/Ada  version  1.6.3.  running  on  a  Sun  4 
workstation  under  UNIX  and  OpenWindows.  At  the  time  of  evaluation,  prices  for  Logis¬ 
cope  started  at  $14,000. 

Logiscope  consists  of  several  parts: 

•  Analyzer.  Processes  source  code  to  provide  the  data  needed  for  the  Results  Editor. 

•  Results  Editor.  Takes  the  results  file  produced  by  the  Analyzer  and,  potentially,  the 
trace  file  produced  by  an  instrumented  program  to  generate  various  reports. 

•  Formatter.  Compacts  the  execution  trace  produced  by  an  instrumented  program. 

•  Static  Archiver.  Gathers  various  analysis  results  and  manages  results  obtained  for  dif¬ 
ferent  versions  of  the  software. 

•  Dynamic  Archiver.  Accumulates  results  for  a  set  of  test  runs  and  enables  multiple  test 
suite  management. 

While  the  Analyzer  is  unique  to  a  particular  programming  language,  the  remaining  tools 
are  language  independent.  All  tools  operate  in  both  interactive  and  batch  mode. 

The  Analyzer  operates  in  either  static  or  dynamic  mode,  although  application  of  Logi¬ 
scope  begins  with  static  analysis  of  the  software  under  test  The  software  should  have  pre¬ 
viously  been  compiled  and,  where  several  compilation  units  are  employed,  these  must  be 
submitted  in  the  compilation  order  (this  restriction  applies  to  Ada  code  only).  In  static 
mode,  the  Analyzer  calculates  the  appropriate  set  of  basic  counts  that  will  be  used  to  as¬ 
sesses  the  quality  of  the  software  under  examination.  In  dynamic  mode,  it  instruments 
source  code  for  instruction  block,  decision-to-decision  path,  LCSAJ  coverage,  or  proce- 
dure-to-procedure  coverage.  Files  are  instrumented  individually  and,  potentially,  for  differ¬ 
ent  types  of  coverage  measurement.  Dynamic  analysis  also  provides  path  and  condition 
identification  to  aid  test  data  generation.  After  instrumentation,  the  user  compiles,  links, 
and  then  executes  instrumented  source  code  as  usual. 

In  general,  the  Analyzer  can  analyze  files  singly  or  as  a  group.  It  generates  a  Results 
File  that  the  Results  Editor  uses  to  generate  a  variety  of  reports.  There  is  a  facility  for  com¬ 
bining  separate  Results  Files  together  to  form  a  single  file  for  a  subsystem,  or  system.  It  can 
search  for  such  items  as  code  based  on  keywords,  or  code  that  falls  within  certain  values 
for  a  given  metric  or  criteria. 
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The  Results  Editor  also  operates  in  static  and  dynamic  modes,  presenting  results  at  dif¬ 
ferent  levels:  details  for  each  application  component,  a  synthesis  of  component  results  for 
the  entire  application,  and  global  application  architecture  information. 

Quality  analysis  is  the  primary  static  analysis  function  and  Logiscope  employs  the 
RADC  quality  metrics  model  to  define  quality  measurement  at  three  levels  of  abstraction. 
At  the  lowest  level  of  the  model  there  are  thirty  five  predefined  pr  imitive  metrics.  The  user 
can  define  upper  and  lower  bounds  for  these  metrics  to  allow  Logiscope  to  flag  out-of- 
bounds  values.  (Verilog  provides  default  values  for  these  bounds  that  are  based  on  their  ex¬ 
perience  over  time.)  The  user  can  specify  algorithms  to  weight  and  combine  the  primitive 
metrics  into  up  to  fifteen  composite  metrics.  Then  higher-level  quality  criteria  allow  clas¬ 
sifying  components  based  on  their  computed  quality  values.  These  criteria  can  also  be  used 
to  get  an  overall  quality  value  for  a  module  and  report  on  final  acceptance  or  rejection  based 
on  this  value. 

Logiscope  distinguishes  between  unit-level  metrics  and  architectural  metrics.  In  the 
first  case,  McCabe’s  control-oriented  measures  are  calculated,  as  well  as  Halstead’s  textu- 
ally-oriented  Software  Science  measures.  At  the  architectural  level,  Logiscope  uses  Mo- 
hanty’s  metrics  to  calculate  accessibility,  testability,  hierarchy  complexity,  structural 
complexity,  system  testability,  call  graph  entropy,  and  the  number  of  direct  calls. 

Quality  results  are  displayed  using  the  Results  Editor  in  static  mode.  In  addition  to  the 
Results  File  produced  by  the  Analyzer,  the  editor  requires  a  Reference  File  that  contains  the 
definitions  of  the  metrics  being  used.  (A  different  Reference  File  can  be  maintained  for 
each  project,  allowing  customization  across  development  efforts.)  For  quality  reporting  at 
the  component  level,  the  user  can  request  Kiviat  diagrams  to  show  achieved  metrics  values 
with  respect  to  the  defined  limit  values.  These  diagrams  are  used  to  display  up  to  30  user- 
selected  metrics,  graphically  showing  those  metrics  that  fall  out-of-bounds.  Metrics  can  be 
displayed  by  component,  or  as  a  statistical  average  over  a  group  of  components.  Kiviat  di¬ 
agrams  can  also  be  segmented  into  quadrants  to  provide  an  additional  layer  of  abstraction. 
Criteria  graphs  are  available  to  display  information  relative  to  all  associations  between  met¬ 
rics  and  criteria,  while  showing  the  situation  of  metrics  with  respect  to  limit  values.  These 
graphs  also  specify  the  category  to  which  the  component  belongs. 

At  the  global  level,  histograms  of  metrics  distributions  and  criteria  distributions  are 
available.  Additionally,  when  there  is  a  large  number  of  components,  the  user  can  request 
a  graphical  distribution  for  a  particular  interval  or  a  distribution  of  components  as  a  func¬ 
tion  of  the  limit  values  defined  in  the  quality  model. 
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Finally,  a  Quality  Report  uses  the  components’  classification  based  on  the  quality  cri¬ 
teria  to  present  a  summary  in  the  form  of  the  percentage  of  components  within  the  set  of 
limit  values.  This  report  assesses  whether  quality  recommendations  for  a  given  criteria 
have  been  met,  or  computes  a  statistical  average  over  a  group  of  components. 

Also  in  static  mode,  the  Results  Editor  generates  control  graphs  to  provide  insight  into 
component  structure  and  behavior,  and  call  graphs  to  describe  the  calling  relationships  of 
analyzed  components.  Control  graphs  can  be  annotated  with  either  source  or  pseudocode 
line  numbers.  Logiscope  supports  control  graph  exploration  with  a  zoom  capability  and  the 
display  of  a  reduced  or  structured  form  of  a  control  graph.  The  reduced  form  can  be  used 
to  verify  that  a  program  meets  the  requirements  of  structured  programming  and  identify  el¬ 
ements  that  do  not  conform.  The  principle  of  control  graph  reduction  consists  of  represent¬ 
ing  as  a  single  node  the  control  structures  that  have  only  one  input  and  one  output.  The  most 
deeply  nested  structures  are  reduced  at  each  successive  reduction  stage,  and  the  user  can 
terminate  this  process  when  desired.  Alternatively,  the  structured  view  displays  the  under¬ 
lying  structures  expressed  in  combinations  of  if-then-else  statements  and  branch  statements 
to  reveal  the  hidden  structuring  of  the  processing.  Measurements  of  a  set  of  intrinsic  char¬ 
acteristics  are  available  for  initial,  reduced,  and  structured  control  graphs.  This  allows  com¬ 
paring  the  set  of  alternative,  equivalent  views  of  a  complex  control  graph  and  can  help  a 
user  to  determine  how  to  improve  the  program  structure. 

Exploration  of  call  graphs  is  also  provided  to  support  the  identification  of  critical  com¬ 
ponents  at  the  architectural  level,  and  of  design  rules  that  have  been  violated.  This  is 
achieved  by  display  of  partial  views  and  manipulation  of  call  graphs,  and  quality  evalua¬ 
tion.  A  call  graph  can  be  displayed  from  any  root,  and  the  display  limited  to  a  view  of  the 
root’s  descendants,  ascendants,  or  both.  Components  can  be  grouped  to  clarify,  for  exam¬ 
ple,  which  components  can  call  that  set  A  call  graph  can  be  limited  to  display  of  the  Logi¬ 
scope  analyzed  components  alone. 

Before  the  Results  Editor  can  be  used  in  dynamic  mode  for  coverage  reporting,  the  trace 
file  produced  by  the  instrumented  program  must  be  formatted.  Subsequently,  the  editor  can 
report  on  the  achieved  coverage  at  both  component  and  global  levels.  At  unit  level,  the  user 
can  request  detailed  reports  on  instruction  block,  decision-to-decision  path,  and  LCSAJ 
coverage.  For  each  type  of  coverage  this  includes  a  listing  identifying  each  instance  of  the 
instruction  block,  decision-to-decision  path,  or  LCSAJ  unit,  supported  by  the  conditions  re¬ 
quired  to  execute  that  instance  as  appropriate,  and  whether  or  not  it  was  executed.  A  path 
list  also  indicates  program  paths  that  have  not  been  executed.  Unit  coverage  results  can  be 
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annotated  on  dynamic  control  graphs  to  provide  easy  assessment  of  the  completeness  of 
unit  testing.  Histograms  of  the  distribution  of  components  as  a  function  of  coverage  rate  are 
available  for  rapid  assessment  of  coverage  progression  throughout  testing.  These  histo¬ 
grams  are  accompanied  by  a  distribution  list  that  shows  the  coverage  achieved  for  each 
component.  This  distribution  list  shows  the  number  of  times  each  test  case  exercised  each 
coverage  instance  and  can  be  used  to  determine  how  well  particular  test  cases  support  or 
duplicate  each  other. 

The  Dynamic  Archiver  is  used  to  group  the  results  obtained  for  a  series  of  tests  to  allow 
reporting  on  cumulative  test  coverage.  Here  formatted  trace  files  are  grouped  into  named 
test  suites  that  are  stored  in  archive  files.  The  Results  Editor  can  then  be  run  on  an  archive 
file  to  generate,  for  example,  distribution  histograms  for  the  accumulated  instruction  block, 
decision-to-decision  path,  and  LCSAJ  coverage  for  all  components. 

At  the  global  level,  the  editor  reports  on  procedure-to-procedure  path  coverage.  Here 
coverage  results  can  be  annotated  on  call  graphs  to  provide  quick  insight  into  the  complete¬ 
ness  of  integration  testing.  An  accompanying  textual  report  details  the  calling  and  called 
relationships  for  each  procedure-to-procedure  path  and  whether  that  path  was  executed.  An 
additional  report,  the  coverage  table,  identifies  the  particular  paths  invoked  by  each  test 
case. 


16.2  Observations 

Ease  of  use.  The  Results  Editor  provides  on-line  help  with  a  list  of  available  commands 
and  command  descriptions.  Components  can  be  grouped  into  a  workspace  to  facilitate  op¬ 
erating  on  a  set  of  components  as  a  whole.  A  broad  selection  of  graphical  output  formats  is 
available,  including  histograms,  tables,  and  pie  charts. 

Logiscope  provides  the  user  with  considerable  flexibility  in  defining  the  quality  char¬ 
acteristics  that  should  be  assessed  and  reported.  It  comes  with  a  series  of  default  quality 
models,  one  for  each  of  five  different  programming  language.  These  can  be  used  as  is,  the 
user  can  tailor  them  to  his  needs,  or  develop  his  own  quality  model  from  scratch. 

Documentation  and  user  support.  The  documentation  is  extensive  and  easy  to  follow. 
Verilog  provided  excellent  user  support. 

Instrumentation  overhead.  Full  instrumentation  of  the  Ada  Lexical  Analyzer  Gener¬ 
ator  (all  components  except  U_support)  gave  a  size  increase  of  just  over  50%. 
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Ada  restrictions.  Pragmas  are  not  processed  by  the  Analyzer.  It  is  not  possible  to  mea¬ 
sure  the  coverage  of  a  terminate  alternative  in  a  selective  wait. 

Problems  encountered.  The  Analyzer  reported  an  error  when  analyzing  one  compo¬ 
nent  (ll_support)  of  the  Ada  Lexical  Analyzer  Generator  and  this  prevented  instrumenting 
this  component  Initially,  some  problems  were  encountered  with  reporting  on  LCSAJ  cov¬ 
erage. 


16.3  Planned  Additions 

Version  3.2  of  Logiscope  with  a  Common  OSF/MOTIF  graphical  user  interface  was  re¬ 
leased  in  fall  1992.  This  new  version  is  menu  driven  and  supports  navigation  between 
source  code  (or  pseudocode)  and  graphs.  Multiple,  integrated  windows  are  simultaneously 
available  to  provide  a  user  with  multiple  perspectives  of  a  single  software  component.  In 
this  new  version,  Logiscope  is  integrated  with  DocBuilder  to  provide  automatic  documen¬ 
tation  of  new  or  existing  code.  Meanwhile,  Verilog  is  working  to  integrate  Logiscope  with 
various  configuration  management  tools. 

A  companion  tool  which  focuses  on  data  flow  analysis  rather  than  control  flow  analysis 
is  under  development 


16.4  Sample  Outputs 


Figures  16-1  through  16-28  provide  sample  outputs  from  Logiscope. 
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Begin 

2  ADAatatenent (  a ) ; 

While  LOW  /-  HIGH  Do 
1  ADA_atateaent (a) ; 

If  ITEM  <  LLSTMBOLTABLE( MIDPOINT)  .KEY  Then 
1  ADA_ata tenant ( a ) ; 

Elaif  not  (XTDt  <  LLSTMBOLTABLE (MIDPOINT) .  KET)  and  (ITEM  - 
LLSTMBOLTABLE ( MIDPOINT ) . KET )  Then 

If  LLSTMBOLTABLE (MIDPOINT) .KIND  -  WHICH  Then 
Exit  of  Subprog ran; 

Elae 

Exit  of  Subprogram;. 

End  If; 

Elae 

1  ADA_atatenent ( a ) ; 

End  If ;~ 

End  of  While; 

End; 


Text  of  component: 

LL_COMPILE/LLFXND  t LL STRINGS i LLSTTLE : return : INTEGER 

Application :  all_erc 
Veraion :  VI 

Language :  ADA 

File  :  ll_oompila.a 


Figure  16-2.  Loglscope  Textual  Representation  of  Control  Graph  of  Function  LLFIND 
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Basic  counts  of  component  : 
LL_CQMPILE/LLFIND : LLSTRINGS : LLSTYLE : return : INTEGER 


|  Number  of  statements 

1  11 

|  Number  of  labels 

I  0 

|  Total  number  of  operators 

|  29 

|  Total  number  of  operands 

1  28 

|  Total  number  of  calls 

|  0 

|  Number  of  comments  |  2  | 
|  Number  of  jump  statements  |  0  j 
|  Number  of  different  operators  |  13  | 
|  Number  of  different  operands  |  II  | 
|  Number  of  different  calls  |  0  | 


Operators 

1  Nbr  . 

Operators 

|  Nbr 

(  )  exp 

j  4 

m 

I  2 

(  )  tab 

1  3 

ELSE 

!  2 

+ 

j  3 

ELSIF  THEN 

1  1 

/ 

i  1 

IF  THEN  . .  END  IF 

i  2 

/- 

|  1 

RETURN 

j  3 

i  5 

WHILE  LOOP  . .  END  LOOP 

|  1 

< 

|  1 

i 

Operands 

|  Nbr 

Operands 

|  Nbr 

0 

1  2 

LLSYNBOLTABLE ( . } .KIND 

|  1 

1 

I  3 

LLTABLESI8E 

1  1 

2 

I  1 

LON 

j  4 

HIGH 

i  4 

MIDPOINT 

|  7 

ITIM 

I  2 

WHICH 

|  1 

LLS YMBOLTABLE ( . ) .KEY 

|  2 

1 

► 


Figure  16*3.  Logiscope  Basic  Counts  for  Function  LLFIND 
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21  With  LL_DECLARATIONS,  INTEGER_TEXT_IO ,  TEXT_IO; 

22 

23  procedure  LL_COMPILE  is 

161  function  LLFIND  (ITEM  i  LLSTRINGS ;  WHICH  :  LL STYLE)  return  INTEGER  is 


162 

—  Find  item  in  symbol  table  —  return  index  or 

0  if  not 

found. 

163 

—  Assumes  symbol  table  is  sorted  in  ascending  order. 

164 

165 

LOW,  MIDPOINT,  HIGH  i  INTEGER; 

166 

167 

begin 

(*  DDP 

1  Begin  *) 

168 

. 

169 

LOW  s-  1; 

170 

HIGH  LLTABLESIZE  +  1; 

171 

while  LOW  /-  HIGH  loop 

(*  DDP 

2  While  •) 

172 

MIDPOINT  (HIGH  +  LOW)  /  2; 

173 

if  ITEM  <  LLSYMBOLTABLE  (MIDPOINT) .KEY  then 

(*  DDP 

3  If  *) 

174 

HIGH  <-  MIDPOINT; 

175 

elsif  ITEM  -  LLSYMBOLTABLE  (MIDPOINT) . KEY  then  (• 

DDP  4  Else  • 

176 

(•  DDP 

5  Else-If *) 

177 

if  LLSYMBOLTABLE  (MIDPOINT) . KIND  -  WHICH 

then  (* 

DDP  6  If  • 

178 

return  (MIDPOINT); 

179 

else 

(*  DDP 

7  Else  • ) 

180 

return  (0); 

181 

end  if; 

182 

else 

(*  DDP 

8  Else  • ) 

183 

—  ITEM  >  LLSYMBOLTABLE  (MIDPOINT)  .KEY 

184 

LOW  t-  MIDPOINT  +  1; 

185 

end  if; 

186 

end  loop; 

(*  DDP 

9  End-While  *) 

187 

return  (0); 

188 

—  item  is  not  in  table 

189 

190 

end  LLFIND; 

191 

192 

procedure  LLPRTSTRING  (STR  t  LLSTRINGS)  is 

193 

—  print  non-blank  prefix  of  atr  in  quotes 

194 

195 

begin 

(*  DDP 

1  Begin  •) 

196 

197 

POT  ( STAND ARD_ERR0R , 

198 

for  I  in  STR' range  loop 

(*  DDP 

2  For_Loop*) 

199 

exit  when  STR.(X)  «  ' 

(•  DDP 

3  If  ’  *) 

200 

(•  DDP 

4  Else  *) 

201 

POT  (STANDARD_ERROR,  STR  (I)); 

202 

end  loop; 

(•  DDP 

5  End~For_Loop* 

203 

POT  ( STAND ARD_ERR0R ,  '■'); 

204 

205  end  LLPRTSTRING; 

852  end  LL_CCMPILE; 

853 


Figure  16-4.  Logiscope  Commented  Listing  for  Function  LLFIND 
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Figure  16*5.  Logiscope  Klvlal  Graph  of  Function  LLFIND 
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Figure  16-6.  Logiscope  Criteria  Graph  of  Function  LLFIND 
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Figure  1 6-7.  Logiscope  Klvlat  Graph  of  All  Components 
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Figure  16-8.  Logiscope  Overall  Metrics  Distribution  for  Program  Length 


Figure  16-9.  Loglscope  Overall  Metrics  Distribution  for  Cyclomatlc  Complexity 
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Categori  | 

Components 

|  Value 

1 

es  | 

1 

1 

LL_COMPILE/LL_TOKBNS 

L1_SUPP0RT 

LL_COMPILE 

LL_SUPPORT/LOOK_AHEAD i LLATTRIBUTE 
: return : LLATTRIBUTE 
LL_COMP I LE/LLMA  IN 

LL_COMPILE/LL_TOKENS/ADVANCE/LOOK 

_AHEAD 

LL_SUPPORT/ALTERNATE/MBRGE_RANGES 
: LLATTRIBUTE i LLATTRIBUTE i return  s L 
LATTRIBUTE 

LL_SUPPORT/CQMPLETE_PATTERNS 

llIsupport/imit_patterk_name i PILE 

_TTPE : LL ST RINGS 
LL_COMPILE/LLPRTTOKEN 
LL_SUPPORT /CONCATENATE : LLATTRIBUT 
E : LLATTRIBUTE i return i LLATTRIBUTE 

LL_SUPPORT/COMPLETE_PAT : LLATTRIBU 
TE 

LL_CONPILE/LLMAIN/READGRAM/BUILDR 
IGHT : INTEGER 

LL_SUPPORT/COHPLETE_PAT/COMPLETE_ 
ALT/RESTRICT : LLATTRIBUTE : SELECTXO 
N_SET 1 return i LLATTRIBUTE 

LL_SUPPORT/CONPLETE_PAT/COMPLETE_ 
ALT/RESOLVE_AMBIGUITY i LLATTRIBUTE 
LL_SUPPORT/EMT_SCAN_PROC 
LL_SUPPORT/EMIT_SCAN_PROC/EHIT_PA 
TTERN_MATCH t  LLATTRIBUTE  >  LLSTRINGS 
: BOOLEAN i BOOLEAN  :  BOOLEAN 
LL_CONPILE/LLTAKEACTION : INTEGER 


90.90 


List  of  components  per  metrics  category 

Application i  all_aro 
Version:  VI 

Language:  ABA 

Metric:  Number  of  statements 

Components :  66 
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Figure  16*11.  Logiscope  Overall  Criteria  Distribution  for  Testability 
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Figure  16-13.  Logiscope  Quality  Report 
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Quality  nodal  definition 


f  Quality  Criteria  Definition  ® 

*MC* 

<  The  aim  of  this  table  is  to  explain  how  the  components 
4  are  classified  for  the  TESTABILITY  criterion 


TESTABILITY  j 

VG 

MAX_LVLS 

N_I0 

I 

associated  diagnosis 

1 

| 

OK 

OK 

OK 

1 

((4+4+2)/10) 

* 

100 

- 

100 

ACCEPTED  | 

OK 

OK 

1 

(<4+4+0)/10) 

« 

100 

- 

80 

TO_STRUCTURE| 

OK 

OK 

1 

( (4+0+2)/10) 

* 

100 

- 

60 

TO_CUT  j 

OK 

1 

( ( 4+0+0 )/10) 

* 

100 

- 

40 

a  1 

OK 

OX 

1 

(( 0+4+2 )/10) 
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Figure  16-14.  Loglscope  Excerpt  from  Default  Quality  Model 
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Figure  16*14  continued:  Logiscope  Excerpt  from  Default  Quality  Model 
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Figure  16-15.  Logiscope  IB  Coverage  of  Function  LLFINO 
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Figure  16-16.  Loglscope  DDP  Coverage  of  Component  BUILDRIGHT 
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Figure  16-16  continued:  Loglscope  DDP  Coverage  of  Component  BUILDRIGHT 
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Figure  16-17  continued:  Logiscope  LCSAJ  Coverage  of  Component  BUILDRIGHT 
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Figure  16*17  continued:  Loglscope  LCSAJ  Coverage  of  Component  BUILDRIGHT 
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Figure  16>18.  Logiscope  IB  Coverage  Histogram 
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Figure  16-19.  Logiscope  DDP  Coverage  Histogram 
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Figure  16-21.  Logiscope  Overall  DDP  Coverage  for  Input  testl.lex 
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Metrics  table  of  root: 

LL_SUPPORT/EMIT_PATTERN_NAME : FILE_TYPE : LLSTRINGS 

Application :  all_arc 
Version:  VI 

Language:  ADA 


Figure  16-22.  Logiscope  Metrics  Table  of  Root 
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Application:  all_arc 
Version:  VI 

Language :  ADA 


Figure  16-23.  Logiscope  Call  Graph  Path  Testability  of  Root 
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Figure  16-24.  Logiscope  Call  Graph  Component  Accessibility  of  Root 
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Version :  VI 

Language :  ADA 


Figure  16-25.  Logiscope  Call  Graph  Calling/Called  Components  of  Root 
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Application:  all_arc 
Version:  VI 

Language:  ADA 


Figure  16-27.  Logiscope  List  of  Call  Graph  Components  per  Level  from  Root 
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Figure  16-28.  Logiscope  PPP  Coverage  of  Root 
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17.  MALPAS 

MALPAS  comprises  a  suite  of  static  analyzers  that  provide  control  flow,  data  use,  in¬ 
put/output  dependency,  and  complexity  analysis.  It  is  unique  among  the  examined  tools  in 
providing  symbolic  execution  and  compliance  analysis  of  code  against  a  formal  specifica¬ 
tion. 


17.1  Tool  Overview 

MALPAS  was  developed  in  the  late  1970s  at  the  United  Kingdom  Ministry  of  Defense 
Royal  Signals  and  Radar  Establishment  to  verify  avionics  and  other  safety-critical  defense 
system  software.  Since  1986  it  has  been  marketed  and  supported  by  TA  Consultancy  Ser¬ 
vices,  Ltd.,  formerly  called  Rex,  Thompson  &  Partners  (RTP).  MALPAS  has  50  users,  in¬ 
cluding  5  Ada  sites.  The  Ada  translator  is  a  relatively  new  product  released  in  July  1991. 
RTP  also  markets  seminars  to  introduce  potential  customers  to  MALPAS  and  training 
courses.  A  user  group  is  supported.  MALPAS  is  available  on  VAX/VMS  platforms.  The 
tools  examined  in  this  study  were  MALPAS  Release  5.1,  EL  Version  5,  Pascal-IL  Transla¬ 
tor  3.1,  and  Ada-IL  Translator  1.01.  The  price  for  MALPAS  and  the  Ada-IL  translator  at 
the  time  was  £60,000. 

The  analyses  performed  by  MALPAS  are  intended  to  assure  software  safety,  reliability, 
consistency,  and  conformance  to  standards.  They  include  the  following: 

•  Control  flow  analysis  to  reveal  the  underlying  program  structure  and  unreachable 
code. 

•  Data  flow  analysis  to  detect  uninitialized  variables  and  successive  assignments  with¬ 
out  an  intervening  use. 

•  Information  flow  analysis  to  identify  input-output  dependencies. 

•  Path  assessment  to  produce  a  structural  complexity  measure. 

•  Partial  analysis  using  program  slicing  to  reduce  analysis  time  for  semantic  and  com¬ 
pliance  analysis. 

•  Semantic  analysis  to  provide  symbolic  execution  for  each  loop-free  path. 

•  Compliance  analysis  to  verify  code  against  formal  specifications. 

MALPAS  analyses  are  based  on  an  Intermediate  Language  (IL)  representation  of  pro¬ 
gram  specifications  or  source  code.  Translators  from  several  languages  (including  Ada,  C, 
Fortran,  and  Pascal)  to  IL  are  available.  The  approach  of  using  a  common  intermediate  lan¬ 
guage  for  analyses  simplifies  the  extension  of  MALPAS’s  capabilities  to  other  program- 
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ming  languages.  Formal  program  specifications  can  also  be  expressed  in  IL.  At  present  no 
automated  translation  tools  for  other  formal  specification  languages  such  as  OBJ,  Vienna 
Development  Method  (VDM),  or  Z  are  supported. 

Analyzing  application  source  code  is  a  two-step  process.  First  the  code  is  translated  into 
IL.  Since  the  Ada  translator  was  not  available  when  the  tool  examination  started,  the  Pascal 
translator  was  examined  first.  Pascal  code  is  translated  as  a  single  complete  program;  this 
is  a  straightforward  process.  The  translation  of  Ada  source  code  to  IL  is  significantly  more 
complicated.  The  sample  Ada  code  analyzed  contained  several  separately  compiled  pack¬ 
ages  and  subunits.  First  the  generic  input/output  packages  used  by  the  program  had  to  be 
instantiated  (by  hand),  translated,  and  loaded  into  an  IL  code  library.  Then  each  program 
unit  had  to  be  translated  and  loaded  into  the  IL  code  library. 

The  second  step  is  to  run  the  analyses  on  the  IL  code.  A  single  tool  controls  all  of  the 
available  analyses.  Options  are  selected  by  command  line  parameters  and  results  are  writ¬ 
ten  to  files  that  can  be  printed.  Default  parameter  settings  for  initial  analyses  of  new  code 
were  set  up  to  include  control  flow,  data  use,  and  information  flow  analyses.  Control  flow, 
data  flow,  and  information  flow  analyses  are  fairly  standard  static  analysis  techniques. 
Structured  programming  has  largely  eliminated  control  flow  anomalies.  Data  flow  and  in¬ 
formation  flow  anomalies,  however,  are  still  useful  indicators  of  potential  problems.  Infor¬ 
mation  flow,  for  example,  identifies  all  of  a  subprogram’s  inputs  and  outputs,  which  may 
be  more  than  those  passed  as  parameters. 

The  compliance  and  semantic  analyses  are  computationally  more  complex.  The  partial 
analysis  capability  allows  these  analyses  to  be  restricted  to  particular  modules  or  paths 
within  the  program.  MALPAS’s  semantic  analysis  option  provides  symbolic  execution  of 
loop-free  code  segments,  that  is,  for  each  possible  path  through  a  segment,  the  value  of  each 
modified  variable  is  given  as  an  algebraic  expression  in  terms  of  the  input  variables.  This 
provides  valuable  feedback  to  a  programmer  about  the  meaning  of  the  code  and  the  results 
that  will  be  produced  when  the  code  is  executed.  The  compliance  testing  option  uses  this 
same  information  to  check  formally  specified  requirements  that  have  been  added  to  the  IL 
code. 


17.2  Observations 

Ease  of  use.  MALPAS  is  a  batch-oriented  tool  even  though  it  may  be  invoked  interac¬ 
tively.  The  only  user  interaction  is  through  the  set  of  options  that  can  be  selected  from  the 
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command  line.  The  large  number  of  options  may  make  MALPAS  “flexible”  for  expert  us¬ 
ers.  Novice  or  casual  users,  however,  may  have  some  difficulty  controlling  non-default  pro¬ 
cessing. 

Introducing  the  intermediate  language  for  analyses  may  cause  problems  for  some  users. 
All  analyses  and  reports  refer  to  the  IL  version  of  the  program  rather  than  to  the  original 
source  code.  The  mapping  back  to  the  original  code  must  be  done  manually.  The  IL  ap¬ 
proach  may  simplify  extending  MALPAS  to  cover  a  range  of  different  programming  lan¬ 
guages  (by  requiring  only  new  IL  translators),  but  it  imposes  a  level  of  separation  between 
the  actual  source  code  and  the  analyses  that  must  be  compensated  for  by  the  user. 

Translating  Ada  source  code  to  the  intermediate  language  was  found  to  be  somewhat 
more  complicated  than  expected.  The  sample  Ada  code  analyzed  contained  several  sepa¬ 
rate  packages  and  subunits,  and  normally  requires  several  compilation  steps.  The  MALPAS 
Ada  to  IL  translator,  however,  required  several  additional  steps  that  Ada  compilers  either 
do  not  need  or  are  able  to  hide. 

Documentation  and  user  support.  Installation  and  operating  instructions  were  clear, 
thorough,  and  accurate.  Installation  required  simply  editing  sample  command  files  to  name 
local  directories  and  disks.  The  manuals  included  good  examples  and  the  tools  operated  ex¬ 
actly  as  described. 

Ada  restrictions.  Support  for  all  aspects  of  Ada  that  can  be  analyzed  statically  is  the 
vendor’s  eventual  goal,  however,  the  current  MALPAS  tools  support  only  a  subset  of  Ada. 
The  Ada  to  IL  translator  recognizes  all  valid  Ada  code  but  the  translation  to  the  intermedi¬ 
ate  language  is  not  complete.  The  intermediate  language,  for  example,  does  not  include  any 
mechanism  for  concurrency,  so  Ada  tasks  cannot  be  translated.  This  restriction  is  particu¬ 
larly  unfortunate  because  execution-based  testing  of  concurrent  programs  is  often  difficult 
to  control.  Repeating  a  particular  test,  for  example,  might  not  produce  the  same  results  each 
time.  Rigorous  static  analyses  of  potential  task  interactions  would  contribute  significantly 
to  identifying  and  correcting  tasking  problems. 

Translation  of  Ada’s  generic  program  units  is  not  supported.  Generic  units  provide  a 
powerful  mechanism  that  simplifies  programs  and  enhances  reuse.  Ada’s  standard  input 
and  output  facilities,  for  example,  are  defined  in  terms  of  generic  packages.  MALPAS  cur¬ 
rently  requires  manual  instantiation  of  any  required  generic  units. 

Access  types  (pointers)  and  dynamic  storage  allocation  are  not  supported.  Analysis  of 
unconstrained  use  of  pointers,  for  example  to  detect  potential  “dangling”  pointers,  is  virtu- 
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ally  impossible.  A  workaround  for  disciplined  use  of  pointers  for  data  structures  such  as 
linked  lists  is  to  define  abstract  data  types  that  encapsulate  the  pointers.  MALPAS  would 
be  able  to  analyze  application  code  that  used  the  abstract  data  types  since  the  pointers  are 
hidden.  MALPAS,  however,  would  not  be  able  to  analyze  an  implementation  of  the  ab¬ 
straction  that  used  pointers. 

Problems  encountered.  The  MALPAS  tools  performed  as  specified  in  their  documen¬ 
tation.  No  failures  occurred  in  use. 


17.3  Planned  Additions 

Version  V6.0  of  MALPAS  (due  for  release  in  November,  1992)  includes  two  additional 
summary  reports  from  the  Semantic  Analyser.  These  reports  present  key  information  from 
the  standard  Semantic  Analyser  report  in  a  form  that  may  be  easier  to  interpret.  Both  reports 
present  the  conditions  for  and  the  assignments  made  on  each  path  through  each  loop-free 
section  of  code.  The  Paths  Table  report  tabulates  the  conditions  and  the  assignments  made 
to  variables  on  each  path.  The  Transforms  report  lists  each  variable  and  shows  the  condi¬ 
tions  under  which  each  assignment  will  be  made. 


17.4  Sample  Outputs 

Figures  17-1  through  17-6  provide  sample  outputs  from  MALPAS. 
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program  average  (  input,  output  ); 

{  Tbia  program  shares  a  stream  between  two  consumers  by  merging  the  } 

{  processes  and  evaluating  the  result  of  the  second  process  eagerly.  ) 
type  resulttype  -  integer;  (  consumer  process  result  type  ) 

streamelement  -  integer;  {  stream  element  type  ) 
var  conslresult:  resulttype;  {  result  returned  by  consumer  #1  ) 
cons2result :  resulttype;  [  result  returned  by  consumer  12  ) 

{  Stream  operations  ) 

procedure  advance  (var  eos:  Boolean;  var  next:  streamelement,-  more:  Boolean ),- 
const  CR  -  13;  {  Advance  the  actual  input  stream  ) 

var  ch :  char  ,- 
begin 

if  more  then 
if  eof  then 
eos  : -  true 
else  begin 

eos  false; 
if  eoln  then  begin 
readln; 
next  CR 
end 

else  begin 
read(ah) ; 
next  :-  ord(ch) 
end 

end 

end; 

procedure  consume,-  f  Consume  the  input  stream  as  one  process  ) 

var  eos:  Boolean;  {  (count  stream  elements  and  sum  stream  elements)  } 
next:  streamelement; 

begin 

conslresult  0; 

conslresult  :■  0; 

advance ( eos , next , true ) ; 

while  not  eos  do  begin  , 

conslresult  :«  conslresult  +  1/  (  count  stream  elements  ) 

conslresult  :«  conslresult  +  next;  [  sum  stream  elements  ) 

advance (eos , next , true ) 
end; 

end; 

begin 

consume  ,- 

writeln ( ' The  average  of  conslresult: 1, 

'  characters  is  *' ,  chr(cons2result  div  conslresult),  '*.') 

end. 


Figure  17*1.  MALPAS  Sample  Pascal  Code  Illustrating  MALPAS  Analyses1 


1.  Due  to  MALPAS ’s  restrictions  on  analysis  of  Ada  access  types,  the  lexical  analyzer  code  used  as  a 
sample  test  program  could  not  be  thoroughly  analyzed.  To  illustrate  the  reports  that  MALPAS  produces  a 
simple  Pascal  program  was  substituted.  This  program  and  the  MALPAS  analysis  reports  are  shown  in  the  fol¬ 
lowing  figures. 
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Cl]  TITLE  average,' 

taj 

[3]  (  Pascal  to  Malpaa  XL  Translator  -  Release  3.0  ] 

141 

[6]  _INCLUDE/NOLIST  "USR: (ADATEST . PASCALIL30] FIXED. PREAMBLE* 

•••  Including  file  USR: [ADATEST. PASCALIL30] FIXED. PREAMBLE ;1  •»* 

*•*  End  of  file  USR:  (ADATEST. PASCALIL30JFIXED. PREAMBLED 

[7]  _INCLUDE/NOLIST  ■ USR:  (ADATEST. PASCALXL30] TEXT. PREAMBLE* 

•««  Including  file  USR: (ADATEST. PASCALIL30] TEXT. PREAMBLE;!  •»» 

End  of  file  USR : (ADATEST . PASCALXL30] TEXT , PREAMBLE: 1  ••• 


(SI 

[10] 

[11] 

[13] 

[13] 

114] 

[16] 

[17] 

[18] 

[30] 

[31] 

[33] 

[33] 

[34] 

WARNING 

[35] 

[36] 

[37] 

[38] 

1391 

[30] 

WARNING 


CONST  cr  :  integer  -  +13; 

CONST  lit 1 theaverage  :  char-array  -  "The  average  of  "; 

CONST  lit 3 characters  :  char-array  «  *  characters  is  * " * ; 

CONST  lit _ 3  t  char-array  -  ***."; 

[*  result  returned  by  consumer  <3  *] 

(*  Stream  operations  *) 

PROC5PEC  advance ( INOUT  eo a  t  boolean, 

INOUT  next  :  integer, 

IN  more  :  boolean) 

IMPLICIT  ( (•*  XL  Global  Parameter  Section  **  ] 

INOUT  input  :  text); 

no  DERIVES  list  specified  for  procedure  advance 
[*  Advance  the  actual  input  stream  •] 

PROCSPEC  consume 

IMPLICIT  ([*•  XL  Global  Parameter  Section  ••  ] 

INOUT  con sire suit ,  oons2result  :  integer 
INOUT  input  :  text); 

no  DERIVES  list  specified  for  procedure  consume 


•] 


[31] 

[*  Consume  the  input  strei 

■m  as  one  process  *] 

[33] 

[*  (count  stream  elements 

and  sum  stream  elements) 

[33] 

[35] 

NAINSPEC  (INOUT  input  :  text 

[36] 

INOUT  output  :  text); 

[37] 

[38] 

PR0C  advance; 

[40] 

VAR  ch:  char; 

[43] 

«1: 

IF  more  THEN 

[43] 

#3: 

IF  eof  text f input 1  THEN 

144] 

15: 

eos  true 

[45] 

ELSE 

[46] 

«6: 

eos  i«  false; 

[47] 

#7: 

IF  eoln _ text  (input)  THEN 

(481 

•  9: 

text _ reedln ( input ) ; 

[491 

110: 

next  :-  cr 

[50] 

ELSE 

[51] 

*11; 

text _ read _ char ( input , 

eh); 

[531 

*13; 

next  charjpos(ch) 

[53] 

*8: 

ENDIF 

Figure  17-2.  MALPAS  Intermediate  Language  Translation  of  Sample 
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1 54 1 

•  4 : 

END  IF 

C5S| 

<2; 

END  IF 

[561 

♦STOP: 

[SKIP] 

[56] 

♦  END: 

ENDPROC  [ ‘advance* ] 

[57] 

[56] 

PROC  consume; 

[60] 

VAR  eos _ 6:  boolean; 

[61] 

VAR  next _ 6:  integer ; 

[63] 

♦  1: 

conslresult  0; 

[64] 

♦  2: 

cons2result  :-  0; 

[65] 

♦  3: 

[65] 

advance ( eos _ 6 ,  next _ 6 ,  true ) ; 

•«*  HARMING  i 

advance  has  not  been  fully  specified 

[66] 

*4: 

LOOP  [while  loop] 

[67] 

♦  6: 

EXIT  (While  loop]  WHEN  NOT(  NOT  eos _ 6); 

[68] 

17: 

conslresult  conslresult  +  1; 

[69] 

(•  count  stream  eleaents  •] 

[70] 

«8: 

cons2result  :«  conslresult  +  next  6 

[71] 

[•  sum  stream  elements  •} 

[72] 

19: 

[72] 

advance (eos _ 6,  next _ 6,  true) 

«*»  WARNING  : 

advance  has  not  been  fully  specified 

[73] 

*5: 

ENDLOOP  (while  loop] 

[74] 

(STOP: 

[SKIP] 

[74] 

(END: 

ENDPROC  [*consume*] 

[751 

[76] 

MAIN 

[79] 

VAR  conslresult:  integer ; 

[BO] 

(•  result  returned  by  consumer  (1  *] 

[81] 

VAR  conslresult  t  integer,- 

[83] 

(1: 

[83] 

consume ( ) ; 

*•*  HARMING  : 

consume  has  not  bean  fully  spacifiad 

[84] 

♦2: 

text  write ( output ,  lit  1  theaverage ) ; 

[90] 

(STOP: 

[SKIP] 

190] 

(END: 

ENDMAIN 

[93] 

[*••*  WARNING  :  WARNINGS  IN  PASS  1  ...  See  Listing  rile 

[95] 

FINISH 

HARMING  :  Procedure  body  for  tart  oat  has  not  baao  defined 
HARMING  :  procedure  body  for  tart  pane  has  not  bean  defined 

***  HARMING  :  Procedure  body  for  text  wrlteln  has  not  been  defined 
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After  ONE-ONE,  13  nodes  removed. 

No  nodes  with  self-loops  removed. 

Node  Id  No  of  pred.  Succ.  nodes 
♦START  0  I END 

♦END  1 

After  KASAI  (from  ONE-ONE),  No  nodes  removed. 

After  HECHT  (from  ONE-ONE),  No  nodes  removed. 

After  HX  (from  HECHT),  No  nodes  removed. 

After  TOTAL  ( from  HK) ,  No  nodes  removed . 

Control  Flow  Summary 

The  procedure  Is  well  structured. 

The  procedure  has  no  unreachable  code  and  no  dynamic  halts. 
The  graph  was  fully  reduced  after  the  following  stages i 
ONE-ONE,  KASAI,  HECHT,  HK,  TOTAL 
The  graph  was  not  fully  reduced  after  the  following  stages: 
None 


Figure  17-3.  MALPAS  Control  Flow  Analysis  of  ADVANCE 


Key 

mmm 

H  -  Data  reed  and  not  subsequently  written  on  some  path  between  the  nodes 
I  “  Data  read  and  not  previously  written  on  same  path  between  the  nodes 
A  -  Date  written  twice  with  no  intervening  read  on  same  path  between  the  nodes 
U  -  Data  written  end  not  subsequently  read  on  some  path  between  the  nodes 
V  -  Data  written  and  not  previously  read  on  some  path  between  the  nodes 
K  -  Data  read  on  all  paths  between  the  nodes 
N  “  Data  written  on  all  paths  between  the  nodes 
E  -  Data  read  on  some  path  between  the  nodes 
L  -  Data  written  on  same  path  between  the  nodes 


After  ONE-ONE 


From 

node 


TO 

node 


Data  Use  Expression 


♦START  fEND  H 

:  eh 

input 

I 

:  input  non 

V 

:  *08 

input 

V 

t  eh 

•08  1 

X 

t  aor* 

E 

:  eh 

Input 

L 

:  ch  . 

•OS 

Summary  of  Possible 

Errors 

more 

next 

next 

more 

input 


next 


No  errors  detected 


Figure  17-4.  MALPAS  Data  Use  Analysis  of  ADVANCE 
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Information  Flow 


After  ONE-ONE 


l 


From  node  t START 

to  node  I END 

Identifier 

may  depend  on  identifier(s) 

eos 

INs/INOUTs 

:  eos  input 

more 

CONSTANTS 

false 

true 

next 

INs/INOUTs 

input 

more 

next 

CONSTANTS 

cr 

input 

INs/INOUTs 

input 

sore 

ch 

INs/INOUTs 

input 

sore 

VARS /OUTS 

ch 

Identifier 

may 

depend  on  conditional 

eos 

*3 

«1 

next 

1? 

#3 

#i 

input 

17 

#3 

u 

ch 

*7 

<3 

#i 

Summary  of  Possible  Error* 


No  errors  detected 


> 


> 


I 


I 


Figure  17-5.  MALPAS  Information  Flow  Analysis  of  ADVANCE 
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SeBantic  Analysis 


After  ONI- ONE 

>  From  node  :  t START 
To  node  #END 

IP  NOT  (store) 

THEN  MAP 
ENDMAP 

ELSIP  more  AND  eof _ text  (input) 

THEN  MAP 

eos  i-  true; 

ENDMAP 

ELSIP  aore  AND  eoln _ text(input)  AND  NOT(eof _ text (input)) 

THEN  MAP 

eos  : “  false; 
next  13; 

input  : -  readln _ text (input ); 

ENDMAP 

ELSIP  Bore  AND  NOT  (eoln _ text  (input))  AfT"  NOT  (eof _ text  (input)) 

THEN  MAP 

eos  false ; 

next  ; -  char_pos ( read _ text  char ( input  > ) ; 

input  s »  skip _ text _ char  ( input )  ; 

ch  s-  read _ text  chartlnput); 

ENDMAP  ENDIP 


Figure  17-6.  MALPAS  Semantic  Analysis  of  ADVANCE 
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18.  QES/MANAGER 

QES/Manager  is  one  component  of  the  QES/Workbench.  To  fully  understand  the  role 
of  QES/Manager,  it  is  necessary  first  to  look  at  the  other  workbench  component,  QES/Ar- 
chitect.  QES/Architect  is  a  database  system  designed  to  create  and  manage  testing  data.  It 
has  automatic  capture/playback,  test  data  generation,  variable  processing,  and  global  edits. 
Fully  instrumented  testcases  can  automatically  change  or  generate  test  data  via  external 
files,  calculations,  predetermined  ranges,  or  system  responses.  Alternatively,  test  data  can 
be  imported  from  external  sources  such  as  screen  form  builders  or  databases,  or  captured 
from  the  workstation.  Conditional  execution  is  provided.  By  prototyping  test  data,  usable 
testcases  can  be  created  that  provide  a  picture  of  the  user  interface.  These  testcases  can  act 
as  the  specification  and  be  used  to  simulate  the  system  operation.  QES/Manager  is  a  subset 
of  QES/Architect.  (The  full  QES/Architect  product  is  expected  to  be  examined  in  the  near 
future.)  QES/Manager  provides  the  data  management  facility  that  supports  documenting 
test  plans  and  testing  activities.  It  also  provides  for  easy  import  of  ASCII  data  and  export 
of  data  to  automated  test  systems. 

Additional  workbench  components  expected  to  be  released  early  in  1993  include  QES/ 
Qease  for  keystroke  capture/playback,  QES/Programmer  for  automatic  unit  test  design  and 
execution,  and  QES/Expert  that  aids  a  user  in  diagnosing  the  cause  of  a  failure. 


18.1  Tool  Overview 

QES/Manager  is  marketed  by  Quality  Engineering  Software,  Inc.  (QES).  In  addition  to 
quality  assurance  (QA)  products,  this  company  markets  consulting  and  programming  ser¬ 
vices,  specializing  in  showing  customers  how  to  improve  QA  and  testing  practices.  A  hot¬ 
line  support  facility  is  available.  QES/Manager  was  released  in  November  1991  and  has 
over  50  users.  It  is  language  independent  and  runs  on  IBM  PC/AT,  or  compatible  machines, 
under  MS-DOS  3.0  or  higher.  QES/Manager  is  compatible  with  local  area  networks 
(LANs).  It  supports  the  following  test  environments:  DOS,  5250/AS-400  emulation,  3270 
emulation,  asynchronous  communications,  UNISYS,  and  Tandem.  Interfaces  exist  to  sev¬ 
eral  test  execution  tools  such  as  AutoTester. 

The  evaluation  was  performed  on  demonstration  version  2.2  of  QES/Manager  running 
on  a  WIN  TurboAT.  This  demonstration  version  is  fully  functional,  although  limited  in  the 
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number  and  size  of  testcases  that  can  be  specified.  At  the  time  of  evaluation,  QES/Manager 
prices  started  at  $2,500. 

QES/Manager  embodies  a  predefined  test  model.  The  basic  test  items  are  as  follows: 

•  Testcases.  Define  the  basic  unit  of  test  data.  Each  Testcase  is  intended  to  be  an  inde¬ 
pendent,  reusable  testing  element  that  tests  one  logical  operation  or  module. 

•  Test  Drivers.  Group  collections  of  Testcases  so  that  a  Test  Driver  consists  of  a  list  of 
Testcases  to  be  run  in  a  specified  order.  Further  levels  of  grouping  are  available:  a 
Test  Driver  List  can  be  used  to  group  Test  Drivers,  and  Master  Drivers  to  group  Test 
Driver  Lists. 

Together,  Testcases  and  Drivers  form  the  test  plan.  A  map  function  showing  the  developed 
organization  of  Testcases  with  Drivers  is  available  from  QES/Manager. 

Relationships  between  test  items  are  maintained  using  the  standard  nomenclature  as¬ 
sociated  with  databases.  All  named  items  have  a  keyword  option  which  can  be  used  for 
such  tasks  as  searching  and  forming  organizational  groupings.  Testcases,  for  example,  can 
be  classified  by  application  or  type  of  test,  such  as  regression,  acceptance,  or  system  tests. 

When  prototyping  test  data,  the  contents  of  a  Testcase  are  specified  by  user-defined 
templates.  When  data  is  imported  from  another  test  tool,  QES/Manager  will  automatically 
define  the  necessary  templates,  without  user  intervention.  In  essence,  a  Testcase  is  equiv¬ 
alent  to  a  template  with  the  associated  input  (keystrokes)  and  output  (responses).  Access  is 
provided  to  the  sequence  of  keystrokes,  not  just  the  result.  For  example,  once  the  Testcase 
is  created,  the  user  can  view  both  the  sequence  of  keystrokes  and  the  final  entry  format. 
Different  types  of  information  can  be  attached  to  a  Testcase  to  sequence,  modify,  identify, 
and  manage  it.  A  full-screen  editor  is  provided  for  creating  and  modifying  templates.  Tem¬ 
plate  fields  can  be  either  named  or  unnamed,  however,  global  edits  can  be  applied  to  named 
fields.  Default  responses  can  be  specified.  Instead  of  manually  entering  test  data,  the  user 
can  define  converters  that  will  import  data  from  ASCII  text  files  and  transform  it  into  QES/ 
Manager  format. 

Prior  to  using  the  Drivers  to  guide  the  execution  of  tests,  the  user  can  view  actual  se¬ 
quencing  of  test  inputs  and  outputs  to  assess  their  correctness.  In  sequence  mode,  QES/ 
Manager  presents  the  operation  of  the  application  (as  represented  through  its  inputs  and 
outputs)  in  its  natural  flow.  That  is,  it  shows  the  sequence  of  default  responses,  keystrokes 
applied  to  the  default  responses,  and  the  response  of  the  application  to  that  input.  A  simu¬ 
lation  function  provides  a  similar  capability,  although  here  the  user  can  specify  time  delays 
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to  be  applied  to  the  sending  of  keystrokes  and  responses  and  can  manually  control  the  flow 
by  stepping  through  the  simulation. 

The  user  can  request  reports  on  any  test  item.  Essentially,  the  reporting  facility  acts  as 
a  database  query  engine.  Here  again,  the  user  specifies  templates  that  define  a  report  layout 
in  terms  of  cursor  positions.  These  layouts  can  be  saved  for  reuse  and  a  copy-and-paste  fa¬ 
cility  is  provided,  together  with  a  sorting  function. 

QES/Manager  provides  a  limited  set  of  administration  functions.  These  allow  the  sys¬ 
tem  administrator  to  assign  new  users  with  password  and  access  permissions,  or  change  the 
access  privileges  of  existing  users.  An  authority  matrix  displays  each  user’s  rights  for  ac¬ 
cess  to  QES/Manager  functions  and  data. 


18.2  Observations 

Ease  of  use.  QES/Manager  is  a  menu-driven  system,  with  both  mouse  and  keyboard 
navigation.  Listboxes  are  provided  to  show  available  items  for  selection.  QES/Manager  can 
be  customized  to  the  extent  of  defining  database  paths,  hotkeys,  printer  ports,  and  the  use 
of  color  in  the  display  screens.  Conformance  to  IEEE  and  Common  User  Interface  (CUI) 
standard  nomenclature,  interfaces,  and  menus  is  intended  to  facilitate  use  of  all  QES/Work- 
bench  tools. 

The  available  on-line  help  includes  context-sensitive  help,  a  manual  with  hypertext 
links  and  a  print  button,  and  a  notepad  for  the  user  to  add  personal  help  information.  A  tu¬ 
torial  and  on-line  demonstration  is  also  available. 

An  import  capability  provides  for  importing  test  cases  in  the  form  of  ASCII  text  files. 
These  are  converted  into  QES/Manager  format  using  user-defined  templates.  A  similar  ex¬ 
port  capability  is  available. 

Documentation  and  user  support.  The  documentation  provided  with  the  demonstra¬ 
tion  version  of  the  tool  was  fairly  limited.  Although  it  provides  good  guidance  for  stepping 
through  one  example  use  of  the  system,  it  does  not  provide  a  general  overview  of  tool  ca¬ 
pabilities  and  usage.  Installation  was  straightforward. 

Problems  encountered.  QES/Manager  operated  as  described  in  the  documentation. 
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18.3  Planned  Additions 

OS/2  and  Windows  support  is  under  development  and  expected  to  become  available  in 
the  fourth  quarter  of  1992. 

18.4  Sample  Outputs 

Figures  18-1  through  18-3  provide  sample  outputs  from  QES/Manager. 
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QES  DOCUMENTATION’  PROJECTS 


Project  Name:  ONLINE  MANUAL 

Main  Task:  ARCHITECT  MANUAL 
Sub- task:  DESIGN 


DOCUMENTATION  PROJECT  MANAGEMENT 
MANUAL  PROJECTS  FOR  ARCHITECT 
DESIGN  MENU  DOCS 


Menu  Name :  CAPTURE 

New  /  Modify  _  Entry  Name: 

_ How  to..  _ Glossary  _ Index 

done:  _  — 

OA :  _  _  — 


Menu  Name:  BUILD 

New  /  Modify  _  Entry  Name: 

_ How  to..  _ Glossary  _ Index 

done:  _  _  — 

QA:  _  _  — 


Menu  Name:  IMPORT 

New  /  Modify  _  Entry  Name: 

_ How  to..  _ Glossary  _ Index 

done:  _  _  — 

Oft;  _  _  — 


Menu  Name :  EXPORT 

New  /  Modify  _  Entry  Name: 

_ How  to..  _ Glossary  _ Index 

done:  _  — 

QA :  — 

Menu  Name : 

New  /  Modify  _  Entry  Name: 

_ How  to..  _ Glossary  _ Index 

done:  _  — 

QA:  — 

Menu  Name : 

New-/  Modify  _  Entry  Name: 

_ How  to..  _ Glossary  _ Index 

done:  _  _  _ 

DA:  — 

Menu  Name : 

New  /  Modify  _  Entry  Name: 

_ How  to..  _ Glossary  _ Index 

done:  _  _  _ 

QA:  _  —  — 


CAPTUP.E  Menu  Documentation 
_ T.O.C.  _ Main  entry  _ Advert/pr 

BUILD  Menu  Documentation 
_ T.O.C.  _ Main  entry  _ Advert/pr 

IMPORT  Menu  Documentation 
_ T.O.C.  _ Main  entry  _ Advert/pr 

EXPORT  menu  documentation 
_ T.O.C.  _ Main  entry  _ Advert/pr 


_ T.O.C.  _ Main  entry  _ Advert/pr 


_ T.O.C.  _ Main  entry  _ Advert/pr 


_ T.O.C.  _ Main  entry  _ Advert/pr 


Figure  16-1.  QES/Manager  Report  Layout 


> 
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MD: 


ONLINE  manual 
h-  TDL:  Architect  manual 
TD:  QES/ intro 

<—  TC:  ABOUT  QES 
TD:  DESIGN 


TD:  1 

E 


—  TC: 
—  TC: 
—  TC: 
l—  TC: 
TEST 
TC: 
TC: 
TC: 


CAPTURE 

BUILD 

IMPORT 

EXPORT 


RUN 

SCHEDULE 
DISCREPANCY 
I—  TD:  MANAGE 

TC:  KEYWORD 
TC:  TEMPLATE 
TC:  VARIABLE 
TC:  RECOVERY 
TC:  TEST  DRIVER 
TC:  TESTCASE 
TC:  TEST  DRIVER  LIST 
TC:  MASTER  DRIVER 
TD:  SECURITY 

TC:  LOGIN 
TC:  ASSIGN  USERS 
TC:  CHANGE  PASSWORDS 

tTD:  REPORT 
TD:  UTILITY 

TC:  BACKUP/ RESTORE 
TC:  CONFIGURATION 
TC:  DEFINE  TOOLS 
■—  TD:  HELP 

|—  TDL:  Manager  manual 
TD:  QES/ intro 

L-  TC:  ABOUT  QES 
|-  TD:  DESIGN 

-  TC:  CAPTURE 
-  TC:  BUILD 
-  TC:  IMPORT 
I—  TC:  EXPORT 
H  TD:  MANAGE 

TC:  KEYWORD 


TC: 

TC: 

TC: 

TC: 

TC: 

TC: 

TC: 


t—  TD: 


TEMPLATE 
VARIABLE 
RECOVERY 
TEST  DRIVER 
TESTCASE 

TEST  DRIVER  LIST 
MASTER  DRIVER 
SECURITY 

ETC:  LOGIN 

TC:  ASSIGN  USERS 
TC:  CHANGE  PASSWORDS 
TD:  REPORT 
TD:  UTILITY 

TC:  BACKUP/RESTORE 
CONFIGURATION 
DEFINE  TOOLS 


t  TC1 


TC: 

TD:  HELP 
TDL:  Tech  ref  manual 
|—  TD:  FILE  INFO 
•—  TD:  MESSAGE  INFO 


Figure  18*2.  QES/Manager  Map  of  Master  Driver 
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PR08LEM  REPORT  FORM 


Number /Name:  0047(F/C/03)  Process  Key  Problem 


Status: 
Fix/E nh: 
Sever ity : 
Function: 
Product: 
Path: 


A  .Resolved 
B.Fix 

C  .Critical 
O.Runable 
E.OES  Architect 
P03.Testease 


No  further  modifications  needed 
Error  needs  correction 
Must  be  fixed  in  next  release 
Usable  in  Regression  test 


Form  Generated:  Tue  Jul  28  14:50:57 


PROBLEM  RESOLUTION  REPORT 


Resolution  Code  (1-8)  _ 

1  «  Fixed 

2  *  Cannot  Reproduce 

3  *  Fixable,  but  Deferred 

4  ■  Cannot  be  Fixed 


Resolution  Version  _ 

5  ■  Withdrawn  by  Tester 

6  «  Works  to  Specification 

7  *  Disagree  with  Enhancement 

8  »  Enhancement  Excepted 


Problem  Summary 


Problem  Resolution 


Resolved  By  _ _  Date 

Resolution  Tested  By  _  Date 


Figure  18-3.  QES/Manager  Problem  Report 
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19.  SoftTest 

SoftTest  supports  requirements-based  test  analysis  using  cause-effect  graphing.  It  de¬ 
rives  test  conditions  to  guide  the  preparation  of  test  data.  It  also  provides  a  measure  of  test 
adequacy  in  terms  of  the  number  of  testable  functional  variations  for  which  tests  have  been 
specified. 


19.1  Tool  Overview 

SoftTest  was  developed  in  1987  and  is  marketed  and  supported  by  Bender  and  Associ¬ 
ates.  There  are  currently  over  150  users.  The  tool  runs  on  any  IBM  PC,  XT,  AT,  PS2  or 
compatible  platform  under  MS-DOS;  since  it  executes  independently  of  the  software  under 
test,  the  target  or  development  environment  of  the  software  is  not  a  restricting  factor.  Bend¬ 
er  also  markets  project  methodology  guidelines,  consulting  services,  and  training  courses 
on  software  quality  assurance  and  testing.  Interfaces  via  outline  files  of  test  case  descrip¬ 
tions  exist  to  several  capture/playback  tools  including  Automator  QA,  AutoTester,  Gate, 
Microsoft  Test  for  Windows,  SQA:Robot,  Sterling  TestPro,  V-Test,  Workstation  Interac¬ 
tive  Test  Tool  for  OS/2,  XRunner,  and  TestRunner. 

The  version  of  SoftTest  examined  was  Beta  Release  4.0,  running  on  a  Compaq  Deskpro 
386/20.  At  the  time  of  examination,  the  price  was  $2,500. 

SoftTest  automates  a  requirements  analysis  technique  called  cause-effect  graphing,  de¬ 
veloped  at  IBM  in  the  early  1970s.  The  primary  phases  of  analysis  are  as  follows: 

•  Extraction  of  node,  relation,  and  constraint  definitions  from  cause-effect  graphs. 

•  Functional  variation  analysis  to  identify  combinations  of  input  conditions  required  for 
tests. 

•  Test  condition  synthesis  to  consolidate  variations  and  produce  minimal  test  sets  that 
will  exercise  all  the  elementary  functions  specified. 

Before  using  SoftTest,  the  user  must  prepare  a  cause-effect  graph  definition  from  the 
functional  specification  of  the  software  under  test  This  process  starts  with  identifying  the 
set  of  input  conditions  (the  causes)  that  a  program  must  respond  to,  mapping  these  to  the 
set  of  output  conditions  (the  effects)  that  the  program  must  produce.  Unique  names  are  as¬ 
signed  to  each  cause  and  to  each  effect,  called  nodes.  SoftTest  distinguishes  primary  nodes, 
that  is,  those  that  are  basic  inputs  or  final  outputs,  from  intermediate  nodes.  In  particular, 
SoftTest  assumes  that  all  effects  that  are  not  inputs  to  any  other  relationship  (that  is,  prima- 
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ry  effects)  are  observable  and  that  effects  that  are  input  to  other  relationships  are  not  ob¬ 
servable  unless  explicitly  specified  as  being  forced  observable.  This  special  case  of  forced 
observable  allows  intermediate  nodes  to  be  used  to  permit  testing  variations  where  the  re¬ 
sults  of  one  functional  variation  may  be  obscured  by  other  variations.  Relations  between 
nodes  are  specified  in  terms  of  logical  relations  such  as  and  and  or.  Finally,  exclusive,  in¬ 
clusive,  one-and-only-one,  and  requires  constraints  that  restrict  the  invokable  combinations 
of  cause  states  are  identified.  Another  statement,  the  mask  statement,  is  included  in  the  con¬ 
straint  category.  It  is  a  qualifier  that  is  used  when  the  true  or  false  state  of  a  particular  node 
will  render  the  state  of  other  node(s)  to  be  indeterminate;  qualifiers  work  by  causing  spec¬ 
ified  nodes  within  a  test  case  to  be  ignored  under  certain  conditions. 

The  resulting  cause-effect  graph  definition  is  expressed  using  a  declarative,  non-proce¬ 
dural  language  based  on  Prolog.  This  language  includes  a  facility  for  defining  a  data  dic¬ 
tionary  of  node  names  that  is  maintained  independently  of  any  particular  cause-effect  graph 
definition;  this  data  dictionary  can  then  be  imported  as  required.  A  subgraph  facility  pro¬ 
vides  for  partitioning  a  large  specification  into  several  parts. 

Once  the  cause-effect  graph  definition  has  been  prepared,  SoftTest  will  perform  a  Vari¬ 
ation  Analysis  to  identify  all  the  individual  unique  functions  the  software  is  required  to  per¬ 
form.  The  thesis  of  this  approach  to  testing  is  that  although  the  number  of  possible 
combinations  of  input  conditions  may  be  very  large,  a  program  can  be  thoroughly  tested  by 
exercising  this  small  set  of  unique  functional  variations.  Some  functional  variations  may 
not  be  testable  because,  for  example,  it  may  be  physically  impossible  for  certain  combina¬ 
tions  of  input  conditions  to  arise.  These  variations  are  flagged  as  infeasible.  The  Variation 
Analysis  can  be  set  to  report  on  primitive  or  full-sensitized  variations.  In  the  first  case,  only 
primary  nodes  are  included,  whereas  a  full-sensitized  analysis  will  include  all  nodes  that 
impact  a  variation.  Two  measures  of  graph  complexity  are  reported:  (1)  the  number  of  func¬ 
tional  variations  divided  by  the  number  of  primary  causes,  and  (2)  the  number  of  functional 
variations  divided  by  the  sum  of  the  number  of  primary  causes  and  the  number  of  primary 
effects.  These  complexity  measures  yield  high  values  when  inputs  are  combined  in  many 
different  ways  and  low  values  if  inputs  are  used  in  simple  relationships. 

The  cause-effect  graph  definition  can  also  be  input  to  the  Picture  Presentation  phase. 
This  phase  produces  a  pictorial  representation  of  the  cause-effect  graph  showing  nodes, 
their  logical  relations,  and  any  applicable  constraints.  It  is  intended  to  aid  the  user  in  ensur¬ 
ing  that  the  cause-effect  description  accurately  reflects  his  understanding  of  the  specifica¬ 
tion’s  logic. 
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The  Test  Synthesis  phase  generates  the  minimum  set  of  test  cases  that  will  ensure  that 
all  feasible  functional  variations  are  exercised.  Each  test  case  is  given  in  the  form  of  a  test 
specification  that  identifies  the  causes  and  effects  that  should  be  true  and  those  that  should 
be  false  for  this  test  case.  (Prior  to  test  execution,  these  test  cases  will  be  used  to  help  in 
manually  determining  the  actual  test  data  needed.)  Test  cases  can  be  reported  in  three 
forms.  For  each  generated  test  case,  a  compact  listing  identifies  the  invokable  causes(s)  and 
their  state(s)  and  the  observable  effect(s)  and  their  state(s).  An  expanded  batch  listing  sup¬ 
plements  this  with  a  full  node  description  for  each  cause  and  effect.  An  expanded  script  list¬ 
ing  provides  much  the  same  information  but  groups  related  causes  and  foliows  them  with 
their  associated  effects  for  each  test  case.  This  phase  also  generates  a  fault  coverage  and 
test  definition  matrix  that  can  be  used  for  planning  and  tracking  the  test  effort.  The  fault 
coverage  matrix  indicates  the  functional  variations  addressed  by  each  test  case  and  includes 
statistics  that  report  on  the  percentage  of  testable  variations  achieved.  The  test  definition 
matrix  identifies  the  nodes '  '  .ded  in  each  test  case. 

Test  Synthesis  is  not  ,„stricted  to  the  generation  of  new  sets  of  test  cases.  For  example, 
if  a  specification  is  changed.  Test  Synthesis  can  be  used  to  report  coverage  of  the  revised 
functional  variations  achieved  by  an  existing  set  of  test  cases,  or  to  determine  the  new  test 
cases  that  must  be  added  to  an  existing  set  to  fully  cover  these  functional  variations.  As  a 
final  point.  Test  Synthesis  itself  may  result  in  the  identification  of  additional  infeasible  vari¬ 
ations.  These  are  variations  based  on  any  declared  constraints  and  other  relationships  that 
are  found  to  be  indeterminate. 

The  latest  SoftTest  release  includes  the  ability  to  extract  test  documentation.  A  2 167  A 
reporting  facility  produces  a  requirements  to  test  case  traceability  report  that  conforms  to 
DoD-STD-2167A  Section  4.1  requirements.  Additionally,  a  Structured  English  require¬ 
ments  specification  can  be  generated  from  the  SoftTest  input  file. 


19.2  Observations 

Ease  of  use.  SoftTest’s  user  interface  provides  simple  menu-driven  commands  to  ini¬ 
tiate  processing,  review,  and  print  results.  It  is  also  possible  to  invoke  one  of  a  number  of 
third-party  text  editors  from  within  the  tool  so  that  graph  specifications  can  be  modified  and 
analyses  rerun  without  leaving  the  tool.  The  hard  part  is  developing  the  complete  cause- 
effect  graph  definition  for  software  to  be  tested;  this  is  not  a  limitation  of  the  tool,  but  re¬ 
flects  the  difficulty  of  the  underlying  specification  task.  Even  though  the  cause-effect  graph 
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language  is  clear  and  simple,  writing  specifications  in  this  form  requires  some  experience. 
Training  courses  offered  by  the  vendor  may  also  prove  useful. 

SoftTest  can  be  used  interactively  or  in  batch  mode.  A  command  queue  facility  pro¬ 
vides  for  specifying  a  group  of  cause-effect  graph  file  name  specifications  to  which  subse¬ 
quent  processing  will  be  applied. 

Documentation  and  user  support.  The  tool  documentation  and  user  support  were 
quite  good.  Installation  was  simple  and  the  tool  operated  exactly  as  described  in  the  refer¬ 
ence  manual.  Two  tutorials  were  provided — one  that  worked  through  examples  of  how  to 
run  the  tool  and  one  that  discussed  requirements- based  testing  in  more  general  terms. 

Problems  encountered.  SoftTest  performed  as  documented.  No  problems  were  en¬ 
countered  in  its  use. 

19.3  Sample  Outputs 

Figures  19-1  through  19-8  provide  sample  outputs  from  SoftTest. 
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C/I  Graph  Input  fori  C-J5  Graph  for  6.1 1  Lexical  Pattern  notation 

TITLE  'C-I  Graph  for  4. It  Lexical  Pattern  Notation'. 

/•Oraphad  7-13-91  by  Blaina  Bragg*/ 

/•tbli  graph  covers  page  4,  section  4.1*/ 

/•NOTH  NT  Mans  Nonterminal  expression*/ 

/*NOTBt  U  aaans  Raqular_exprassion*/ 

/•NOTH  VB  saand  vertical  Bar  alternative  separator*/ 

MOD1S 

STARTANAL- 'Begin  the  Lexical  Pattern  Notation  Checking ' . 

SC_FOUND- ' A  seal-colon  was  found* 

| ’A  saai-colon  was  not  found'. 

8C_XFROR- '  Display  the  NO  S  INI -COLON  POUND  error  Masage'|/b. 

DIP_LEX_END»'A  saai-colon  was  found -define  as  the  end  of  the  lexicon' |/b  OB8. 

AS_roUND  81- 'The  Assign_symbol  was  found  before  the  end  of  the  lexicon* 

J'The  Assign_symbol  was  not  found'. 

ASXRROR- '  Dipley  the” NO  ASSIGNMENT  SYMBOL  error  ms  sage'  |/b. 

3XG_NT_XVAL— ' Nonterminal  def ined-Begin  Mon_terainal  syntax  check' | /b  OBS. 

cai_XQ_LXT- ’ The  first  character  of  the  Mon_tersLinal  is  a  letter' 

| 'The  first  character  of  the  Non_terainal  is  not  a  letter'. 
CH1IRROR-' Display  the  INVALID  PIXST  NT  CHARACTER  error  Manage'  |/b. 
VALID_CHl-'The  first  character  of  the  Nonterminal  is  valid'  J/b  OBS. 

SOB_CH_VAL- ‘ The  eubseqent  characters  in  the  NT  are  valid* 

| 'One  or  worm  of  the  subsequent  NT  characters  are  invalid' . 
SOB_CHAR_KM- ' Display  the  INVALID  SOBSBQDBNT  CHARACTER! s)  error  Mseage’|/b. 
D0BJ7J-' There  is  a  DOUBLE  UNDERSCORE  in  the  NT  expression' )/b. 

DOB  US  EM- 'Display  the  DOUBLE  UNDRSOORE  error  Msaage'  |/b. 

LL_LT_1LN- ' The  NT  exspression  is  less  than  or  equal  to  om  11m  long’ 

|  'The  NT  expression  is  more  than  om  11m  long'. 

NTjQRJEM-' Display  the  NT  EXPRESSION  XS  TOO  LONG  error  message' |/b. 

VXLID_NT— ' The  Non _tandul  expression  is  valid '  | /b  OM. 

RE_CONT-'Ths  reqular  expression  eonteiM  ou  or  more  characters' 

| 'The  reqular  expression  has  no  charseers  (is  NOLL)'. 

NOLL_RE_EM« ' Display  the  NO  REGOLAR  EXPRESSION  error  amssage'l/b. 

BXGRXEVAL- ■ Begin  the  REGOLAR  EXPRESSION  syntax  evaluation' ( /b  OBS. 

QS_VALID« ' The  Quotation  Syebol  syntax  and  contents  are  valid 'OB8. 

QS- 'Therm  is  om  or  eore  quotations  syabols  in  the  RE' 

| 'There  is  no  quotation  sysblola  in  the  RE'. 

QS_BAL— 'The  quotation  symbols  balance'. 

QSjQX-'Tha  quotation  symbol  syntax  is  OX'OBS. 

QS_OONT>* There  ie  om  or  more  characters  within  each  sat  of  quotes'. 

QS_I1- ' • j/b  OBS. 

QS~I2-'  '  |/b  OBS. 

OB~QS_EM-' Display  the  UNBALANCED  QUOTATIONS  error  ms  sage'  |/b. 

NULL_QS_EMw ’ Display  the  EMPTT  QUOTATION  SYMBOLS  error  message' |/b. 

VB_VALID- ' The  vertical  Bar  syntax  and  contents  are  valid' | /b  OBS. 

VB- ‘There  is  om  or  more  VB  alternative  separators  in  the  RS'  |/b. 
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Figure  19*1.  SoftTest  Graph  Entry  Phase  Input 
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C/I  Graph  Input  fort  C~E  Qraph  t or  4.1 i  Lexical  Pattmrn  notation 

VB_LS_CONT-' There  la  ona  or  aora  charactars  on  tha  left  side  of  aach  VB ' 

| 'Thera  are  no  character  on  tha  laft  aide  of  aach  VB'. 

VB_NS_C0NT« '  Thera  la  ona  or  store  character  a  on  tha  right  alda  of  aach  VB ' . 
VB~X1«' ' I /bOBS . 

VB~I2-’ • I /bOBS. 

VB~L8_IM« ' Olaplay  MISSING  LIFT  SI DM  ALTERNATE  error  Message' |/b. 

VB~KS~*H-' Display  MISSING  BIGHT  SIDI  ALTERNATE  error  sassage' |/b. 

DD_VALID- ' Tha  double  dot  syntax  and  eontanta  are  valid' |/b  OBS . 

DD-' There  la  double  dot  notation  In  tha  IE' 

| 'Thera  la  no  double  dot  notation  In  tha  Rl'. 

DD  _LS_I1»Q- '  The  characters  on  tha  laft  side  are  In  alngle  quotas ' 

| 'The  character  on  tha  laft  alda  are  not  In  alngle  quotas'. 

DD_LS_CONT» ' There  la  ona  or  aora  charactars  on  tha  laft  side  of  tha  DD' 

| 'Thera  a re  no  character  on  tha  laft  alda  of  tha  DD'. 

DD_R8_lNQ*'The  characters  on  tha  right  side  a re  In  alngle  quotas'. 

DD_IS_CONT* •  There  la  one  or  store  characters  on  the  right  aide  of  the  DD*. 
DD~L8~VIL-'The  left  side  of  the  DD  la  VALID ' OBS . 

DD~RS~VAL- ’ Tha  right  side  of  the  DD  la  VALID 'OBS. 

DD~LS~QEM- '  Display  the  MISSING  LIFT  SIDI  QUOTATION  error  ateaeaga'  |/b. 

DD~LS~CEM- ' Display  tha  DD  EMPTY  LIFT  SIDI  QUOTATION  error  Message' I /b. 
DD~IS~QEM«' Display  tha  MISSING  RIGHT  SIDE  QUOTATION  error  Message' |/b. 

DD_RS~CEH« ' Display  the  DD  EMPTY  NIGHT  SIDI  QUOTATION  error  Masaaga' j/b. 

DD~I1«"  |/b  OBS. 

DD~I2-' ' |/b  OBS. 

BBA_VALID“ *  The  Brace a  ayntax  and  contents  are  valid 'OBS. 

BRA- ' There  are  ona  or  More  braces  la  the  II’ 

| 'There  are  no  braces  In  tha  II'. 

8HA_HAL— 'The  Braces  In  the  IB  balance' 

| 'The  Braces  In  tha  RI  do  not  balance ' . 

BRA_OOMT»' There  ara  one  or  nore  characters  within  the  braces' 

| ‘Thera  are  no  character  within  the  braces ' . 

BRAI1-' * |/b  OBS. 

BRA_I2-’ • |/b  OBS. 

BNAJIB JRM- ■ Display  the  UNBALANCED  BRACES  error  awasaga ' . 

8RA_SKPTY_EM« ' Display  the  EMPTY  BRACES  error  Mas sags' . 


BRX_VALID- ' The  Bracket  syntax  and  contents  are  valld'OBS. 
BRX-'Thera  are  ona  or  more  Brackets  la  the  It' 

| 'Thera  are  no  Brackets  In  the  II'. 

8RK_BA»'The  Brackets  In  the  RE  balance' 

| ‘The  brackets  in  tha  II  do  not  balance' . 

8RX_C0MT- ' There  are  one  or  aora  characters  within  the  braces' 

| ‘Thar  ere  no  characters  within  the  braces'. 

BRE_I 1- ' ' |/b  OBS. 

BmTz2-"  j/b  OBS. 

■RIJ®®—***  -'Display  the  UNBALANCID  BRACKETS  error  Message*  |/b. 
BRKEMPTYBM- ' Display  the  EMPTY  BRACKETS  error  Message' J/b. 
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Figure  19-1  continued:  SoftTest  Graph  Entry  Phase  input 
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C/I  Graph  Input  fort  C-S  Graph  tor  4.  It  Lexical  Pattarn  Notation 

VALID_RX« '  Tha  regular  axpraaalon  la  valid' |/b  OBS. 

VALID~LKX" ' Tha  L1XIOON  IS  VALID* |/b  OBS. 

DO_H*XT» ’ Bag in  chocking  tha  naxt  laxleal  atataaant*. 


OOHSTRAXms 

MASK (not  BXG_RK_XVAL,QS,VB,DD, BRA, BRK) . 

MASK (not  DBF~LEX_BHD , AS_FOUHD_BB ) . 

MASK (not  8S0~MT_ivAL,  c5l_IQ_LST) . 

MASK  ( not  VALID_CH1 ,  S0B__CH~VAL ,  DOB_OS ,  LL_LT_1LH  J . 

MASK  (not  VALID JW,M_C0Mt7. 

MASK (not  QS,QS~BAL> 

MASK (not  QS_Ok7qS_COHT ) . 

MASK (not  VbTvBLS^CONT , VBRSCOKT ) . 

MASK (not  DO,  DD_LS_IHQ , DD^LS-CONT , DD_RS_INQ , DD_RS_COKT ) . 

MASK (not  BRA, BRABAL , BRAOOHT ) . 

MASK (not  BRK , BRK_BAL , HRK^COKT ) . 

RZLATIOKS 

DirLKKJEHD I -STARTAKAL  and  SCJTOOm. 

SCKRRORt-not  SC_KOOND  and  STARTAKAL . 

BKOKT^IVAL : -DKf7lKK_BKD  and  AS_rOOKD_BB. 

ASKRROR i -DIFLKXKKD  and  not  ASfOCKD  J« . 
VALIDJSIi-BBOJItYvAL  and  CH1_BQ_LKT.~ 

CHI  JERROR I -HkTkTIVAL  and  not~c»I_KQ_Lrr. 

VALID_MT i -VALID  CHI  and  SOB  CH  VAL  and  not  DOB  OS  and  LL  LT  ILK. 
SUBCHARJtMi -VALIDCH1  and  nOt~S0B_CH_VAL. 

D0B_BSJg;4»-VALID_c5l  and  DUB_OS. 

MTORIKi -VALIDCHl  AMS  not  LL_LT  ILK. 

BBCrSkVAL i -VAL ID  _KT  and  KZ_00Rt7 
H0LL_R8_BMi “VALXD_KT  and  not-Kl_00KT. 

QSOKi-QS  and  Q8JBAL. 

Qs7ll«-<lS_OK  and~QS_COHT . 

Q3_I2 i -not  QS. 

0s7vALIDt-0S_Il  or  QS_I2. 

UBJ>S_*Ki-gS~and  not  QS_BAL. 

KOLL_5b_EM t -flS_OK  and  not  QS_COKT. 

VB_I2i-VB  and  VB_L8_COKT  and  VB  KS  COHT. 

VB^Ui-not  VB. 

VB_LB_BK  i  -VB  and  not  VB_LS_CO*rr . 

VB^KB^BKi-VB  and  not  VB^KB^OOKT. 

VB_VALID i -VB_I 1  or  VB_I2. 

DD_L8_VALi-DD  and  DD_L8_IKQ  and  DD_Lfl_COKT. 

DD^RS^VALi-DD  and  0D~M~iaQ  and  DDJlS^eORT. 

0D~Il7-  DD_LS_VAL  and  DD_KS_VAL. 

DD  I2i-not  DD. 


MftTMt  *.0(MT4  >  MT40-000  OKAPI  SHUT  Phaaa  Input 

Figure  19*1  continued:  SoftTest  Graph  Entry  Phase  Input 


-  3 


19-7 


SoftTest 


PART  II 


11/06/92  03:53p.m.  Di VC0STC*O\inA\LBXXCaB.CBO 

C/S  Graph  Input  fori  C-B  Graph  tor  4.1 i  LaxicAl  Pattern  Vocation  # 

D0_VALID:-DD_I1  or  D0_I2. 

DD~LS_QXM:-DD  and  not~DD_LS_INQ. 

0D  LS_CEM:-DD  and  not  DD_LS_CONT . 

DD~RS~QEM»-DD  and  not  DD~RS_INQ. 
dd~rs~ceMj-dd  and  not  CD  RS  cont. 

• 

BRA_I1:-BRA  and  BRA_BAL  and  BRA_COWT. 

BRA~l2«-not  BRA. 

BRA~VALID i -BRAI 1  or  BRAI2. 

BRA_ OB_*K : -BRA  and  not  BRA_BAL. 

BRA~ SMPTT  KKl-BRA  and  not  IRA  COST. 


BRK_Z1 t-BRK  and  BRK_BAL  and  BRXJ30NT.  • 

BRK~ 12 t-not  BRK. 

BRX~VALID i -BRK_I 1  or  BRX_X2. 

BRX~OB_SKi>BRK~and  not  BRK_HAL. 

BRK_EMPTT_BMt-BRK  and  not  BRX_COHT. 

VALID  RB « -BBO_RE_KVAL  and  QS_VALID  and  VB_VALID  and  DD_VALID  and  BRA_VALID 
and  BRX_VALID.  -  -  - 

VALID _LBX : -VALIDRK . 

DO  NEXT: -VALID  LEX. 


TESTS 


SoftTMt  4.0<KTA  )  #IT*0-000  GRAPE  BMTBT  PAasa  Input 

Figure  19-1  continued:  SoftTest  Graph  Entry  Phase  Input 
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NOTES  <ONTESTABLE>  and  <INFSA5IBLB>  variation*  from  tha  Taat  Synthaaia 
phaaa  HAVE  baan  margad  into  this  raport. 

Functional  Variations  fori 
DEF_LEXJENDt-START_ANAL  AND  SC_FOtJND 

1.  If  START_ANAL  and  SC_FOUND 

than  DEF_LEX_END. 

2.  If  not  START_ANAL 

(and  SC_F00ND) 
than  not” DEF_LBX_END . 

3.  If  not  SC_F00ND 

(and  START_ANAL) 
than  not  DSF  LEX  END. 


Functional  Variations  fort 
SCERRORs -STARTAKAL  AND  not  EC_FOOND 

4.  If  START_ANAL  and  not  SC_FOUND 

than  SCJ5RROR. 

5.  If  not  STARTJUJAL 

(and  not  SC_F0UHD) 
than  not  SC_XRROR. 

6.  If  SC_FOUND  ~ 

(and  STARTJUiAL) 
than  not  SC  ERROR. 


Functional  Variations  fori 
BEG_HT_EVAL«-DEF_LEX_END  AND  AS  FOOND_BX 

7.  If  DEF_LEX_END  and  AS_FOOHD_BE 

than  iEO_NT_EVAL. 

8.  If  not  DEF_LXX_Xm> 

(and  AS_FOOND_BE  MASKad) 
than  not-BEG_NT_EVAL. 

9.  If  not  ASFOONDHK 

(and  DEF_LEX_END) 
than  not  BEG  NT  EVAL. 


Functional  variations  fort 
AS_ERRORi-DEF_LEX_SND  AND  not  ASJTOCND  BE 

~10.~If  DBF_LEX_END~and  not  AS_FOVND_BE 
than  AS_ERROR. 

11.  If  not  DEFJLEX_END 

(and  not” AB_FOOWD_BK  MASKad) 

than  not  as  Error.- 

12.  if  as_found_5e 

(and  DEF_LEX_END ) 
than  not  AS  ERROR. 


SoftTMt  4.0(KTA  )  SET 40-000  VARIATION  AMALTSZI  Phasa  Output  -  1  - 

Figure  19-2.  SoftTest  Variation  Analysis  Phase  Output 
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Functional  Variations:  C-S  Graph  tor  4.1  *  Lexical  Bottom  notation 

functional  Variations  fori 
VALID_CH1 I - BIG_NT_KVAL  AMD  CH1_BQ_LBT 

13. ~If  BEO_NT_SVXL  and  CH1_BQ_LST 

than  VALID_CH1 . 

14.  Zf  not  BSO_NTjrVAL 

(and  chi~eq”l*t  KASKad) 
than  not  VALID_CH1. 

15.  If  not  CH1_XQ_LTT 

(and  BBOjnr“BVAL) 
than  not  VALID  CHI. 


Functional  Variations  fori 
CHlJBRRORl-BSG  NT  CVAL  AND  not  CH1_IQ_L*T 

16.  “if  BXO_MT_XVAL  and  not  CH1_IQ_L*T 

than  CHl“lRftOR. 

17.  If  not  B*Q_NT_EVAL 

(and  not“cHl_XQ_LK  KASKad) 
than  not  CHl_KRROR. 

18.  If  CHI  KQ  UCT~ 

(and” BXG_HT_RVAL) 
than  not  CHl~BRROR. 


functional  Variations  fori 

VALID_KTt -VALID_CH1  AND  SUB_CH_VAL  AMD  LL_LT_1LM  AMD  not  DOBOS 

19.  If  VALID  CHl'and  SOB  CBjni£  and  not  DDB  DS  and 
LL_LT_1LN 

than  VALID_MT. 

20.  If  not  VALID_CH1 

(and  SOB_CH_VAL  KASKad  and  not  DUBJJS  KASKad  and  LL  LT  1LH 
KASKad ) than  not  VALIDJMT. 

21.  If  not  SOB_CH~VAL 

(and  VALID_CH1  and  not  DDB_0S  and  LL_LT_1LN) 
than  not  VALZD_NT. 

22.  If  not  LL_LT_1LN 

(and  VALID_CH1  and  BOB_CH_VAL  and  not  DOBJOS) 
than  not  VALZD_MT. 

23.  If  D0BJT8 

(and~VALID_CHl  and  SOB_CH_VAL  and  LL  LT  1LH) 
than  not  VALID  NT. 


functional  Variations  fori 

SUB_CHAR_KMi -VALID_CH1  AMD  not  80B_CH_VAL 

24.  If  VALID_CH1  and“not  SOB  CH  VAL 

than  Soi_CHAR_IM. 

25.  Zf  not  VALID_CH1 

(and  not  SUB_CH_VAL  MASXad) 
than  not  SOB_CHAR_SM . 

26.  If  SOB  CH  VAL_ 


SoftTMt  4.0(SCTA  )  smo-ooo  VARIATION  ANALYSIS  fhasa  Output 

Figure  19-2  continued:  SoftTest  Variation  Analysis  Phase  Output 
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Functional  Variation*:  C-S  Graph  tor  4 .It  Lmxlcal  Pattern  Potation 

VALID  RX:-BIG_RX_tVAL  AND  QS_VALID  AMD  VB_VALXD  AND  DD_VALXD  AND  BRA  VALID  AND 

brjTvalid 

129.  If  BEC_RE_EVAL  and  QS_VALXD  and  VB_VALXD  and  DD_VALXD  and 
BRA_VALID  and  SriTvALID 

than  VALXD_JU5. 

130.  If  not  BEG_RE_BVAL 

(and  QS_VALID  and  VB_VALXD  and  DDJTALXD  and  BRA  VALID  and 
BRX_VALID ) than  not  VALIDJtt. 

131.  If  not  QS_VALID  “ 

(and  BBO_RE_EVAL  and  VB_VALXD  and  DD_VALXD  and  BRA  VALID 
and  BRX_VALID) 

~  than  not  VALID_RX. 

132.  If  not  VB_VALXD~ 

(and  8XC_RE_EVAL  and  QSJVALXD  and  DD_VALXD  and  BRA  VALID 
and  BRKVALID) 

than  not  VALID_R1. 

133.  If  not  DDJVALID” 

(and  BEOJREJEVAL  and  QS_VALID  and  VB_VALID  and  BRAVALID 
and  BRK_VALID)  ** 

~  than  not  VALID_R*. 

134.  If  not  BRAJFALXD 

(and  BXQ~RE  (VAL  and  Q8  VALID  and  VB  VALID  and  DD  VALID  and 
8RK_VALID ) than  not~ VAL ID_RE . 

135.  If  not  BRKVALID- 

(and  BBO_RX_XVAL  and  Qs  VALID  and  VB  VALID  and  DD  VALID  and 
BRA_VALID ) than  not  VALID  RE. 


Functional  Variationa  fori 
VALXD_LXX l -VALID  JIB 

136?  If  VALID JRE 

than  VALID  LEX. 

137.  If  not  VALID _RE 

than  not  VALID  LEX. 


Functional  Variationa  fori 
DO_MEXT  t -VALID_LEX 

138.  If  VALXDJLEX 

than  DOJNEXT. 

139.  If  not  VALID_LEX 

than  not  DO~NEXT. 


— >  Thara  vara  NO  Xnfaaaibla  variationa. 
— >  Thara  vara  NO  DETESTABLE  Variationa. 


tofttMt  4.0CKTA  )  MttO-QQO  VARIATION  ANALYSIS  Phasa  Output 

Figure  19-2  continued:  SoftTest  Variation  Analysis  Phase  Output 
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Test  Synthesis  Output!  C-E  Graph  for  4. It  Latcloal  Pat  tarn  Notation 

[Format!  NoNodeXAMES  |noRAW[ Streamlined)  |ALLaffeets|EXTeaslvaTCs 
('IT'  •  (yetkaala  of  HEW  taata  specified. ] 

['S'  ■  Expanded- SCRIPT  Taat  Caaaa  r aqua# tad. ] 

TEST  CMS  ll 

Cauaa(a) < 

Bagla  tha  Lax leal  Pattarn  Notation  Chocking 
A  a  ami-colon  was  found 
Bffact(a) i 

A  a ami-colon  waa  found-dafino  a a  tha  and  of  tha  lexicon 
Cauaa(a)! 

Tha  Aaaign_symbol  waa  found  bafora  tha  and  of  tha  laxicon 
Effact(a) i 

Mon_tarminal  dafinad-Bagin  Hontacainal  ayntax  chock 
Caaaa(a)i 

Tha  fir at  character  of  tha  Nonterminal  la  a  latter 
Bf fact (a) i 

Tha  firat  character  of  tha  Nonterminal  la  valid 
Caaaa (a) t 

Tha  aubaaqant  charactara  in  tha  NT  era  valid 
Tha  NT  exapiraaaion  is  laaa  than  or  agual  to  ona  line  long 
Bffaet(a)i 

Tha  Non_tar«inal  axpraaaion  is  valid 
Cauam ( a ) i 

Tha  regular  axpraaaion  contains  ona  or  mors  characters 
Bffaet(a) i 

Begin  tha  REGULAR  EXPRESSION  syntax  evaluation 
Camse(s)i 

There  is  ona  or  more  quotations  symbols  in  tha  RE 
Tha  quotation  symbols  balance 
Effect (a) i 

Tha  quotation  symbol  syntax  is  OK 
Camsa(a) i 

There  is  ona  or  more  characters  within  each  sot  of  quotas 
Effact(s) i 

Tha  Quotation  Symbol  ayntax  and  contents  are  valid 
Causa(s) t 

There  ia  ona  or  more  VB  alternative  separators  in  tha  RE 
Thera  ia  ona  or  more  characters  on  tha  loft  aids  of  aach  VB 
There  is  ona  or  more  charactara  on  the  right  side  of  each  VB 
Effect(a) • 

Tha  Vertical  Bar  syntax  and  contsnts  are  valid 


toftTaat  «.0(KTA  )  fSTM-000  TEST  SYNTHESIS  Phasa  Output  -  X  . 


Figure  19*3.  SoftTest  Test  Synthesis  Phase  Output 
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Tact  Synthaaia  Output!  c-E  Graph  ter  4.1 1  Lexical  Pattern  notation 

Ct«M(l)  • 

Thar*  la  doubla  dot  notation  In  tha  RX 
Tha  charactara  on  tha  laft  alda  ara  In  alngla  quotaa 
Thara  la  ona  or  aora  charactara  on  tha  laft  alda  of  tha  DO 
lffact(a)i 

Tha  laft  alda  of  tha  DO  la  VALID 
Cauaa<a)i 

Tha  charactara  on  tha  right  alda  ara  in  alngla  quotaa 
Thara  la  ona  or  aora  charactara  on  tha  sight  alda  of  tha  0D 
Xf fact (a) i 

Tha  right  alda  of  tha  DD  la  VALID 

Tha  doubla  dot  ayntax  and  contanta  ara  valid 

Cauaa(a) < 

Thara  ara  ona  or  aora  bracaa  in  tha  RZ 
Tha  Bracaa  in  tha  RX  balanca 

Thara  arc  ona  or  nora  charactara  within  tha  bracaa 
Bffact(a) i 

Tha  Bracaa  ayntax  and  contanta  ara  valid 

not  Diaplay  tha  UNBALANCED  BRACES  arror  aaaaaga 

not  Diaplay  tha  EMPTY  BRACES  arror  aaaaaga 

Cauaa(a)! 

Thara  ara  ona  or  aora  Brackata  In  tha  RX 
Tha  Brackata  In  tha  RX  balanca 

Thara  ara  ona  or  aora  charactara  within  tha  braoaa 
Xffact(a) i 

Tha  Brackat  ayntax  and  contanta  ara  valid 
Tha  ragular  axpraaaion  la  valid 
Tha  LEXICON  ZB  VALID 

Bag In  c hacking  tha  naxt  laxical  atataaant 


sour cat  Raw  taat 


(OftTaat  4.0(tCTA  )  MHO-OOO  TBS*  STRTXXSZS  Iklll  Output  -  2  - 

Figure  19-3  continued:  SoftTest  Test  Synthesis  Phase  Output 
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Test  Synthesis  Outputs  C~K  Graph  tor  4. It  Lax  leal  p attorn  Notation 
test  on  its 
Causa (a) s 

Begin  tha  Lax leal  Pattern  Notation  Cheeking 
A  aeei-colon  was  found 
Bffae t(s)s 

A  aeei-colon  was  found-da fine  as  the  end  of  tha  lexicon 
Cauee<e)t 

The  Assign_aymbol  was  found  before  the  and  of  the  lexicon 
Bffeet(s)s 

Nonterminal  defined -Bag in  Non_terminal  syntax  cheek 
Causa(s) i 

The  first  character  of  the  Nonterminal  ie  a  letter 
Xffeet(s)i 

The  first  character  of  the  Xon_tarmlnal  la  valid 
Cause ( s ) i 

The  eubeeqent  characters  in  the  NT  are  valid 
The  NT  exspreesion  is  leea  than  or  equal  to  one  line  long 
Sffect(s) > 

The  Nonterminal  express  ion  is  valid 
Cause (e) i 

The  reqular  expression  contains  one  or  more  characters 
Nffeet(s)i 

Begin  the  REGULAR  EXPRESSION  syntax  evaluation 
Ceuse(a) i 

There  is  one  or  store  quotations  symbols  in  the  NX 
The  quotation  symbols  balance 
Bffeet(s) i 

The  quotation  symbol  syntax  is  OK 
Cause(s)t 

There  is  one  or  more  characters  within  each  set  of  quotes 
Bffeet(s) t 

Tha  Quotation  Symbol  syntax  and  contents  axe  valid 
Cause(s) i 

There  is  one  or  more  VB  alternative  separators  in  the  NX 
There  is  one  or  more  characters  on  the  left  side  of  each  VB 
Thera  is  one  or  more  characters  on  the  right  side  of  each  VB 
Xffaet<s)< 

The  Vertical  Bar  syntax  and  contents  are  valid 
Cause(s)i 

Thera  is  double  dot  notation  in  the  NX 

The  characters  on  the  left  side  are  in  single  quotes 

There  is  one  or  more  characters  on  tha  left  side  of  the  00 


SoftTsst  4.0OCTA  )  AST 40-000  TEST  STXTXXSXS  Phase  Output 
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Figure  19*3  continued:  SoftTest  Test  Synthesis  Phase  Output 
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Tact  Synthasis  Output*  C~E  Graph  for  4.  It  Lexical  Pat  torn  notation 
If fact (a) i 

Tha  left  aida  of  tba  00  la  VALID 
Cauaa(a) i 

Tha  charactara  on  tha  right  alda  ara  In  aingla  quotaa 
There  la  ona  or  aora  charactara  on  tha  right  aida  of  tha  DO 
Kffaot(a) i 

Tha  right  aida  of  tha  DO  ia  VALID 

Tha  doubla  dot  ayntax  and  contanta  ara  valid 

Cauaa(a) i 

Thara  arc  ona  or  aura  bracaa  in  tha  IX 
Tha  Braeaa  in  tha  IX  balanca 

Thara  ara  ona  or  aora  charactara  within  tha  bracaa 
Xffact(a) i 

Tha  Braeaa  ayntax  and  contanta  ara  valid 

not  Olaplay  tha  OMBALANCXD  HACKS  arror  aeaaage 

not  Display  tha  KKPTX  BRACKS  arror  aaaaaga 

Ca«aa(a)i 

Thara  ara  ona  or  aora  Bracket a  in  tha  IX 
Tha  brackets  in  tha  II  do  not  balanca 
Thara  ara  ona  or  aora  characters  within  tha  bracaa 
■ffact(s)i 

not  Tha  Bracket  ayntax  and  contanta  ara  valid 
Display  tha  ORBALABCXD  BRACKXTS  arror  aassaga 
not  Begin  chocking  tha  next  lexical  atataaant 


Sourest  Maw  test 


SeftTaat  4.MKTA  )  STTW-000  TEST  RXtBfll  Phase  Output  .  ]j  . 

Figure  19-3  continued:  SoftTest  Test  Synthesis  Phase  Output 
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Tut  Synthaaia  Output  i  C-X  Graph  tor  4.  It  Laxloal  Pa  team  Notation 


— >  For  n  *  25  Prinary  Cauaaa,  than 

— >  2*n  -  33,554,432  THBOKKTICAL  XazisuzB  toolbar  of  Taat  Caaaa. 

— >  SoftTaat  ganaratad  18  Taat  Caaaa,  which  yialda  a 
— >  1,864,135  to  1  Taat  Caaa  Coaprassion  Katie. 

— >  SoftTaat  ganaratad  139  Functional  variationa,  which  yialda  a 
— >  8  to  1  Functional  Variationa  to  Taat  Casa  Coapraaaloa  Katio. 

— >  Taat  Synthasia  Xlapaad  Tiawi  7  Minutas 


SoftTaat  4.0<aCTA  >  MTM'OOO  SBSZ  SYKTIXSIS  Phaaa  Output 


-  S3  - 


Figure  19-3  continued:  SoftTest  Test  Synthesis  Phase  Output 
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TMt  Synthaaia  output i  C-S  graph  tax  4.  It  Lexical  Pattern  Potation 


I'U*  ■  tyrrOinli  of  MEU  inti  apaclflad.) 

tatt  Cass  ti.  Punctional  Variation  COVKXMZ  IBTIII 


Figure  19-4.  SoftTest  Functional  Variation  Coverage  Matrix 
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di  \co*TCEo\xaji\unxeoH.Mx 


Tout  Synthuaia  Outputs  OS  Graph  tor  4. It  Xaxleal  Pattarn  Potation 


for  n  »  25  frlaary  Cauaaa,  than 
2‘n  •  S3.SS4.432  THEORETICAL  Naxlu  lhadoar  af  Taat  Caaaa. 

toftTaat  (anaratad  II  Taat  Caaaa,  which  ylalda  a 
— >  1,964,135  to  1  Zaat  Caaa  Coapraaaion  Ratio. 

— >  toftTaat  (anaratad  139  Functional  varlatfena,  which  ylalda  a 

— >  a  to  1  Puuctloaal  Variations  to  Saat  Caaa  Coopgo salon  Ratio. 

••>  Tast  tynthaala  Elapsad  Tlaa:  7  Nlnutaa 


toftTaat  A.OtltTA  )  mu-900  TEST  9TOXHS9X9  7  ha  SO  Output 

Figure  1 9*5  continued:  SoftTest  Test  Case  vs.  Node  Name  Definition  Matrix  • 
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Figure  19-6.  SoftTest  Cause-Effect  Graph 
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Figure  19-6  continued:  SoftTest  Cause-Effect  Graph 
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Functional  Requirement  Raport  Fila 
Filename <  Oi\CUSTCEG\IDA\LEXICON.DOC 

Thia  docuaant  waa  extracted  from  SoftTest -generated  data  using 
tha  following  fila  and  format  specifications* 
D:\CDSTCEG\IDA\LEXIGON 

Craatad/Laat  Hodifiadt  11/06/92  04t01  p.m. 

Format i  MoModaMAKBS | NoRAW [ Streaml inad  ] |ALLaffacta|BXTanaivaTC 
'll*  ■  Synthaaia  of  NEW  taata  apacifiad. 

'S'  »  Expandad-SCRIPT  Test  Caaaa  raquaatad. 


[  NOTE*  Thia  docuaant  waa  craatad  using  "SoftTest ’*  a  Computar 
Aaaiatad  Softwar a  Engineering  product  from  Bandar  6  Aaaociataa 
Inc.,  Larkspur,  California.  SoftTaat  ganarataa  taat  caaa  output 
baaad  on  uaar-providad  functional  raquiramanta  specifications . 

Baaad  on  thaaa  spacif icstions,  tha  following  Spacification  Docuaant 
baa  baan  prepared. ) 


Functional  Bpacifleations  fort  C-I  Graph  for  4. It  Lexical  Pattern  Rotation 


1.  IF  Begin  tha  Lexical  Pattern  Rotation  Checking 

AND  A  oeat-colon  waa  found 

THEM  a  aaai-colon  was  found-dafina  as  the  and  of  the  lexicon. 

2.  ZF  Begin  tha  Lexical  Pattern  Notation  Checking 

AND  A  aaai-colon  was  not  found 
THRU  Display  tha  no  BENI-COLOR  PODRD  error  aaaaaga. 

3.  IP  A  aaai-colon  was  found-dafina  as  tha  and  of  tha  lexicon 

AND  Tha  Aselgn_syabol  waa  found  before  the  end  of  tha  lexicon 
THEN  Non_terninal  defined-Begln  Nonterminal  syntax  cback. 

4.  IP  A  aaai-colon  waa  found-dafina  aa  tha  and  of  tha  lexicon 

AND  Tha  Aasign_ayabol  waa  not  found 
THEN  Diplay  tha~NO  ASSIGNMENT  SYMBOL  error  aaaaaga. 

5.  IP  Nonterminal  defined-Begin  Non_t*rminal  syntax  check 

AND  Tha  firet  character  of  tha  Non_tara*inal  is  a  latter 
THEN  Tha  first  character  of  tha  Nonterminal  is  valid. 

6.  ZF  Nonterminal  defined-Begin  Nonterminal  syntax  check 

AND  Tha  first  character  of  tha  Nonterminal  is  not  a  latter 
THEN  Display  tha  INVALID  FIRST  NT  CHARACTER  error  aessaga. 

7.  IF  Tha  first  character  of  tha  Ron  terminal  is  valid 


leftTest  4.0(SCTA  )  MUO-OOO  Functional  Raquiramant  Raport  Fila 


-  1  - 


Figure  19*7.  SoftTest  Functional  Requirements  Report 
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Functional  Requiresient  Report  file 

30.  IF  (There  are  one  or  more  Bracket!  in  the  R1 

AND  The  Bracket!  in  the  RB  balance 

AND  There  are  one  or  more  character!  within  the  bracee] 
OR  (There  are  no  Bracket!  in  the  RX] 

TRIM  The  Bracket  syntax  and  contanta  are  valid. 

31.  ZF  Thera  are  one  or  eore  Bracket!  in  the  RB 

AND  The  bracket!  in  the  RB  do  not  balance 
THEM  Diaplay  the  UNBALANCED  BRACKETS  error  message. 

32.  IF  There  ere  one  or  more  Bracket!  in  the  RX 

AND  Thar  are  no  character!  within  the  brace! 

THEM  Display  the  EMPTY  brackets  error  me  a a age. 

33.  ZF  Begin  the  REGULAR  EXPRESSION  syntax  evaluation 

AMD  The  Quotation  Syebol  syntax  and  contanta  are  valid 
AND  The  Vertical  Bar  syntax  and  contanta  are  valid 
AMD  The  double  dot  syntax  and  contents  are  valid 
AMD  The  Bracea  syntax  and  contents  are  valid 
AMD  The  Bracket  syntax  and  contents  are  valid 
THSM  The  regular  expraasion  is  valid. 

34.  ZF  The  regular  expression  ie  valid 

THEN  The  LEXICON  ZB  VALID. 

35.  ZF  The  LEXICON  IS  VALID 

THEM  Begin  checking  the  next  lexical  statement. 


In  addition,  the  following  constraints  Bust  be  applied 
to  the  above  specifications! 

1.  WHEN  MOT  Begin  the  REGULAR  EXPRESSION  syntax  evaluation 

THEM  the  following  condition (s)  ere  IEDXTXRKZMATX t 

There  ie  one  or  eore  quotations  symbols  in  the  RX 

There  ie  one  or  eore  VB  alternative  separators  in  the  RX 

There  ie  double  dot  notation  in  the  RX 

There  are  one  or  no  re  braces  in  the  RX 

There  are  one  or  more  Brackets  in  the  RX 

2.  WHEN  MOT  A  semi-colon  was  found-define  ae  the  end  of  the  lexicon 

THEM  the  following  oondition(s)  are  INDETERMINATE i 

The  Aaslgn_symbol  was  found  before  the  end  of  the  lexicon 

3.  WHEN  MOT  Nonterminal  defined-Begin  Nonterminal  syntax  check 

THEN  the  following  condltlon(s)  are  INDETERMINATE i 

The  first  character  of  the  Non  terminal  is  a  letter 


toftlMt  4.mkta  )  #*140-000  Functional  Requirement  Report  Pile  -  4  - 

Figure  19-7  continued:  SoftTest  Functional  Requirements  Report 
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Functional  Raquiraaant  Raport  Fila 

4.  WHIM  MOT  Tha  first  charactar  of  tha  Mon_tarainal  ia  valid 

THEN  tha  following  condition(s)  ara  INDETERMINATE : 

Tha  aubaaqant  charactara  in  tha  NT  ara  valid 

Thara  ia  a  DOUBLE  UNDERSCORE  in  tha  NT  axpraaaion 

Tha  MT  axapraaaion  ia  laaa  than  or  aqual  to  ona  lina  long 

5.  WHEN  rot  Tha  Non_tarminal  axpraaaion  ia  valid 

THEN  tha  following  eondition(a)  ara  INDETERMINATE : 

Tha  raqular  axpraaaion  containa  ona  or  aora  charactara 

6.  WHEN  Thara  ia  no  quotation  ayablola  in  tha  Rl 

THEN  tha  following  condition (a)  ara  INDETERMINATE t 
Tha  quotation  ayabola  balanco 

7.  WHEN  mot  Tha  quotation  ayabol  syntax  la  OK 

THEM  tha  following  condition(a)  ara  INDETERMINATE i 

Thara  ia  ona  or  aora  charactara  within  aach  aat  of  quotaa 

8.  WHEN  MOT  Thara  ia  ona  or  aora  VB  altaraativa  aaparatora  in  tha  RE 

THEM  tha  following  condltlon(a)  ara  INDETERMINATE t 

Thara  ia  ona  or  aora  charactara  on  tha  laft  alda  of  aach  VB 
Thara  ia  ona  or  aora  charactara  on  tha  right  alda  of  aach  VB 


Thara  ia  no  doublo  dot  notation  in  tha  EE 
THEM  tha  following  condition(a)  ara  INDETERMINATE > 

Tha  oharactara  on  tha  laft  aida  ara  in  aingla  quotaa 
Than  ia  ona  or  aora  charactara  on  tha  laft  aida  of  tha  DD 
Tha  charactara  on  tha  right  aida  ara  in  aingla  quotaa 
Thara  ia  ona  or  aora  charactara  on  tha  right  aida  of  tha  dd 

10.  MHSM  Thara  ara  no  braeaa  in  tha  RE 

THEM  tha  following  condition(a)  ara  INDETERMINATE ■ 

Tha  Bracas  in  tha  RE  balanco 

Thara  ara  ona  or  aora  charactara  within  tha  braeaa 


11. 


Thara  ara  no  Brackata  in  tha  RE 
THEM  tha  following  condition(a)  ara  INDETERMINATE s 
Tha  Brackets  in  tha  RE  balance 

Thara  ara  ona  or  aora  charactara  within  tha  braeaa 
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U1M1MI  Dj\CTJSTC30\IDA\LXXIOQII.TXT 

Thia  docuaant  taaplata  was  aatractad  froa  BoftTaat-gonaratad  data 
using  tha  following  fila  and  foraat  opacifications! 

Dl \CCSTCSO\XDA\X£XXCON 

Craatad/Laat  Modified:  11/06/92  04i01  p.a. 

roraatt  MoModaMAMSf  |  MoKAM  ( Straaal  lnad )  |  ALLaf facta  |  SXTana  lraTC 
•It'  -  Synthaaia  of  HIM  taata  apaelfiad. 

'S'  -  Expanded-SCRIPT  Taat  Caaaa  raqaaatad. 


4.1  C-S  Orapb  for  4. It  Lexical  Patton  notation 

(  mnti  Thia  dominant  was  eraatad  using  ■SoftToaft  a  Computer 
Xaaiatad  Softwaro  Engineering  product  froa  Bandar  |  Aaaooiataa 
Inc.,  Larkspur,  California.  SoftTest  generates  taat  oaaa  output 
based  on  user-provided  functional  roquiranants  apaeifleationa. 

It  in  inpar stive  that  111  of  tha  taata  apacifiad  by  SoftTaat  ba 
auceaoafully  run  naing  tha  SAMS  version  of  any  progran  nodule (s) 
undar  taat  in  ordar  to  aaaura  that  full  functional  coverage  ia 
achieved.  ] 


> 


> 


» 


» 
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4.1.1  TUT  CASK  LEXICOM-01 

This  la  a  functional  taat  caaa  intended  to  daaonatrata  that  tba 
requirements  llatad  In  tha  naxt  auction  parfom  correctly. 


4. 1.1.1  LEXICON-01  REQUIREMENTS  TRACEABILITY 

Thin  tact  caaa  will  taat  tha  following  functional  requirewentai 

1.  IF  Bag in  tha  Lexical  Pattern  Notation  Chocking 
ABO  A  semi-colon  waa  found 

THEN  A  a awl-colon  waa  found -define  aa  tha  and  of  tha  lexicon. 


2.  IF  Begin  tha  Lexical  Pattern  Notation  Checking 

AND  A  a  awl-colon  waa  not  found 
THEN  Dlaplay  tha  NO  BEMI-COLON  FOUND  error  weaeage. 

3.  IF  A  aanl -colon  waa  found-da  fine  aa  tha  and  of  tha  lexloon 

AND  The  Aaalgn_ayabol  waa  found  befora  tha  and  of  tha  lexicon 
THEN  Nonjtarwlnal  deflned-Begln  Nonterminal  ayntax  check. 

4.  IF  A  aawl -colon  waa  found-daflna  aa  tha  and  of  tha  lexicon 

AND  Tha  Aaaign_ayabol  waa  not  found 
THEN  Diplay  tha  wo  assxonkent  STKBOL  error  WI 


5.  IP  Non_tanalnal  daflnad-Bagln  NOn.tarwinal  ayntax  check 

AND~Tha  fir at  character  of  tha_Non_tarnlnal  la  a  latter 
THEN  Tha  firat  character  of  tha  Nonterminal  la  valid. 

6.  IF  Non_tazmlna 1  daflnad-Bagln  Non_taxnlnal  ayntax  check 

AND  Tha  firat  character  of  the  Non_ternlnal  la  not  a  letter 
THEN  Dlaplay  tha  INVALID  FIRST  NT  CHARACTER  error  weaeage. 

7.  IF  The  firat  character  of  the  Hon_torelnal  la  valid 

AND  The  eubeeqant  characters  In  the  NT  are  valid 
AND  The  NT  exspreaalon  la  lea a  than  or  equal  to  one  line  long 
AND  NOT  There  la  a  DOUBLE  UNDERSCORE  In  tha  NT  expression 
THEN  Tha  Nonjtarwlnal  expression  la  valid. 

B.  IF  The  first  character  of  the  Nonterminal  la  valid 

AND  One  or  acre  of  tha  subsequent  NT  characters  are  Invalid 
THEN  Dlaplay  tha  INVALID  SUBSEQUENT  CHARACTERS )  error  weaeage. 

9.  IF  The  first  character  of  tha  Non_t*rwlnal  la  valid 

AND  Thera  la  a  DOUBLE  UNDERSCORE  la  tha  NT  expression 
THEN  Display  the  double  UNDRSCORS  error  was sage. 

10.  IF  The  first  character  of  the  Noajtarwlaal  la  valid 
AND  The  NT  expression  is  wore  than  one  line  long 
THIN  Display  tha  NT  EXPRESSION  IS  too  L0N0  error  weaeage. 


t  FUo 
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35.  IF  [Thar*  trt  on*  or  Bora  bracaa  in  tha  RX 

AMD  Tha  Bracaa  in  tha  RX  balance 

A MS  Thara  ara  ona  or  Bora  eharaetara  within  tha  braoaa] 
Oil  (Thara  ara  no  bracaa  in  tha  U) 

T8DI  Tha  Bracaa  syntax  and  contents  ara  valid. 

36.  ZP  Thara  ara  ona  or  aora  bracaa  in  tha  RX 

AMD  Tha  Braces  in  tha  RX  do  not  balance 
THEM  Display  tha  UNBALANCED  BRACKS  error  message. 


4. 1.1. 2  LXXIOOH-01  INITIALIZATION 


4. 1.1. 3  LXXIOOM-Ol  TEST  I SPOTS 

1.  Begin  the  Lexical  Pattern  Notation  Checking 

2.  A  seal-colon  was  found 

3.  Tha  Asaign_eymbol  was  found  before  tha  and  of  tha  lexicon 

4.  Tha  first  character  of  tha  Nonterminal  is  a  latter 

5.  Tha  subsagent  characters  in  tha  NT  ara  valid 

6.  Tha  NT  exapreaaion  is  lass  than  or  equal  to  ona  line  long 

7.  Tha  reqular  expression  contains  ona  er  norm  characters 

8.  Thara  la  ona  or  an  re  quotations  eyabola  in  tha  RX 

9.  Tha  quotation  symbols  balance 

10.  Thara  is  ona  or  aora  characters  within  each  sat  of  quotas 

11.  Thara  is  ona  or  aora  VB  alternative  separators  in  tha  RX 

12.  Thara  is  one  or  aora  characters  on  tha  loft  side  of  each  VB 

13.  Thara  is  one  or  aora  characters  on  tha  right  side  of  each  VB 

14.  Thara  is  double  dot  notation  in  tha  RX 

15.  Tha  characters  on  tha  loft  side  ara  in  single  quotas 

16.  Thara  is  one  or  aora  characters  on  tha  left  side  of  tha  DD 

17.  Tha  characters  on  tha  right  aids  ara  in  single  quotas 

18.  Thara  is  one  or  aora  characters  on  the  right  side  of  the  DD 

19.  Thara  ara  ona  or  aora  braces  in  tha  RX 

20.  Tha  Braces  in  tha  RX  balance 

21.  Thara  ara  one  or  aora  characters  within  tha  braces 

22.  Thara  are  ona  or  aora  Brackets  in  tha  RX 

23.  Tha  Brackets  in  tha  RX  balance 

24.  Thara  ara  ona  or  aora  characters  within  tha  braces 


4. 1.1. 4  LBXZO0N-01  BXPXCTBD  TXST  RESULTS 

1.  A  saai-eolon  was  found -define  as  the  and  of  tha  lexicon 

2.  Nonterminal  dafined-Begin  Nonterminal  syntax  check 
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3.  Tha  fir at  charactar  of  tha  Mon_tarainal  is  valid 

4.  Tha  Non  tanainal  axpraaelon  ia  valid 

5.  Begin  tha  REGULAR  EXPRESSION  syntax  evaluation 

6.  Tha  quotation  symbol  syntax  is  OK 

7.  Tha  Quotation  Symbol  syntax  and  eontants  a ra  valid 

8.  Tha  Vartieal  Bar  syntax  and  eontants  ara  valid 

9.  Tha  left  slda  of  tha  DD  ia  VALID 

10.  Tha  right  alda  of  tha  DO  ia  VALID 

11.  Tha  double  dot  syntax  and  eontants  ara  valid 

12.  Tha  Braeas  syntax  and  eontants  ara  valid 

13.  BOT  Display  tha  UNBALANCED  braces  error  message 

14.  NOT  Display  tha  BMFTY  BRACKS  error  aaaaaga 

15.  Tha  Braekat  syntax  and  eontants  ara  valid 

16.  Tha  regular  expression  is  valid 

17.  Tha  LEXICON  IS  VALID 

18.  Begin  checking  tha  next  laxleal  statement 


4. 1.1. 5  LEXXC0M-01  CRITERIA  FOR  EVALUATING  RESULTS 


4. 1.1. 6  LEXZCQR-01  TEST  PROCEDURE 


Cause(a)i 

Begin  tha  Lexical  Pattern  Rotation  Checking 
A  seal-colon  was  found 
Effact(a) i 

A  s amt -colon  ms  found -da  fine  as  tha  and  of  tha  lexicon 
Cauaa(s)i 

Tha  Assign_ayabol  was  found  before  tha  and  of  tha  lexicon 
Effect(s)< 

Nonterminal  definad-Begin  Ron_t«aiul  syntax  check 
Causa(a)i 

Tha  first  charactar  of  the  Ron_t*rainal  is  a  latter 
Effoct(s)i 

The  first  character  of  tha  lleii_tuBiMl  is  valid 
Causa(s)i 

Tha  subseqent  eharaetars  in  tha  RT  are  valid 
Tha  NT  exsprasaion  is  lass  than  or  equal  to  one  line  long 
Effect (a) i 

Tha  Non_termin*l  expression  is  valid 
Cauaa(s) t 

Tha  regular  expression  contains  one  or  more  eharaetars 
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If tact (a) i 

Bag in  tha  REGULAR  EXPRESSION  syntax  avaluation 


Cause(s) t 

Thara  la  ona  or  mora  quotations  aymbola  in  tha  RB 
Tha  quotation  aymbola  balance 
Iffact(a) t 

Tha  quotation  symbol  syntax  is  OX 
Cauaa(a) i 

Thara  ia  ona  or  mora  characters  within  aach  sat  of  quotas 
Bffact(a) t 

Tha  Quotation  Symbol  syntax  and  contants  are  valid 
Causa(s) i 

Thara  is  ona  or  mora  VB  alternative  separators  in  tha  RB 
There  is  ona  or  mora  characters  on  tha  left  side  of  aach  VB 
Thara  is  one  or  mora  characters  on  tha  right  side  of  aach  VB 
Bffeet(e) i 

Tha  Vertical  Bar  syntax  and  contents  ara  valid 
Causa (s) t 

There  is  double  dot  notation  in  tha  RB 
Tha  characters  on  tha  left  aide  ara  in  single  quotas 
Thara  is  ona  or  mora  characters  on  tha  left  side  of  tha  DO 
Bffact(a) t 

Tha  left  aide  of  tha  00  is  VALID 


Causa (a) i 

Tha  characters  on  tha  right  side  ara  ia  single  quotas 
Thara  is  ona  or  mora  characters  on  tha  right  side  of  tha  DO 
Effect (a) i 

Tha  right  side  of  tha  DO  is  VALID 

Tha  double  dot  syntax  and  contents  ara  valid 

Cause(s) i 

Thara  are  ona  or  mora  braces  in  tha  RB 
Tha  Braces  in  tha  RB  balance 

Thara  ara  ona  or  more  characters  within  tha  braces 
Bffact(s) i 

The  Braces  syntax  and  contents  ara  valid 

not  Display  tha  ORBALAXCSD  BRACES  error  massage 

not  Display  tha  8MFTT  BRACES  error  massage 

Cause(a)i 

Thara  are  ona  or  more  Brackets  in  tha  RB 
Tha  Brackets  in  tha  RB  balance 
There  are  one  or  mors  characters  within  tha  braces 
Bffact(s)t 

Tha  Bracket  syntax  and  contents  ara  valid 
Tha  regular  expression  is  valid 
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Tha  LEXICON  IS  VALID 

Bag  in  c  hacking  tha  naxt  laxical  atataaant 
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20.  SQA:  Manager 

SQA:Manager  is  a  management  information  and  decision  support  system  for  software 
testing.  Essentially  SQA:Manager  provides  a  cataloging  system  for  test  documents  and  in¬ 
ventory  items,  and  a  tracking  system  to  record  incidents  and  problems  found  during  the  test 
process.  Problem  data  is  used  to  forecast  quality  metrics  such  as  reliability  and  failure  in¬ 
tensity.  Cost  data  is  used  to  report  on  the  cost  of  testing  and  the  cost  of  repair.  The  test  pro¬ 
cess  supported  by  SQA:Manager  is  based  on  a  standard  IEEE  test  methodology.  As 
marketed,  the  tool  can  generate  test  reports  according  to  the  associated  IEEE  standards,  or 
to  conform  with  the  appropriate  U.S.  Government  standards.  The  user  can,  however,  tailor 
existing  formats  and  create  new  report  formats. 

SQA:Robot  is  a  companion  capture/playback  and  comparison  tool  for  MS-Windows 
and  OS/2  PC  environments.  It  writes  test  procedures,  test  cases,  test  case  results  and  inci¬ 
dents  directly  into  the  SQA:Manager  database. 


20.1  Tool  Overview 

SQA:Manager  is  marketed  by  Software  Quality  Automation.  Based  on-site  needs  as¬ 
sessment  audits,  this  organization  also  develops  custom  testing  tools.  A  user  hot-line  is 
available.  SQA:Manager  has  been  available  since  1990,  and  has  over  100  users.  It  is  lan¬ 
guage  independent  and  runs  on  IBM  PC/AT,  or  compatible  machines,  under  MS-DOS  (ver¬ 
sion  3.0  or  higher)  with  MS-Windows.  It  also  runs  under  OS/2  and  UNIX.  A  network 
version  that  operates  on  Novell’s  Netware  and  3COM  local  area  network  product  is  avail¬ 
able.  The  tool’s  database  implementation  is  based  on  Borland  International’s  Paradox  com¬ 
mercial  product,  a  relational  database  in  conformance  with  IEEE  recommendations.  The 
export  capabilities  of  this  database  can  be  used  to  convert  SQA:Manager  data  for  use  with 
other  PC  applications  such  as  Lotus  1-2-3.  Paradox  reporting  functions  can  also  be  used  to 
extend  those  of  SQArManager. 

The  evaluation  was  performed  on  an  evaluation  copy  of  SQA.Manager  version  2.0  run¬ 
ning  on  a  WIN  TurboAT.  At  the  time  of  evaluation,  the  price  of  SQA:Manager  was  reduced 
from  $3,500  to  $995. 

SQA:Manager  provides  a  cataloguing  system  for  test  documentation  and  test  inventory 
items.  It  defines  an  entity-relationship  model  for  the  software  components  under  test  with 
documentation  and  other  data  generated  in  the  testing  process  indexed  to  this  model.  Thus, 
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SQAiManager  operates  on  a  database  of  test-related  information  specific  to  a  software 
product  being  tested.  (Multiple  database  are  allowed  so  that  a  separate  database  can  be  set 
up  for  individual  products.)  The  database  maintains  an  inventory  of  reusable  test  resources, 
specifically  test  cases,  test  procedures,  and  testing  tools.  The  specific  types  of  test  objects 
recognized  are  as  follows: 

•  Test  Plan.  Details  the  entire  testing  process,  that  is,  the  items  to  be  tested  and  how, 
the  resources  to  be  used,  and  the  timeframe  within  which  the  testing  is  to  occur.  It 
provides  the  links  between  software  components,  test  cases,  and  test  procedures. 

•  Test  Design  Specification.  Details  the  ways  in  which  a  software  feature  is  to  be  tested 
and  identifies  the  specific  tests  to  be  used. 

•  Test  Procedure.  The  detailed,  step-by-step  instructions  for  test  setup  and  operation, 
and  evaluation  of  test  results. 

•  Test  Cycle.  A  series  of  tests  within  a  release  of  software. 

•  Test  Case.  A  test  case  is  a  set  of  test  data  designed  to  achieve  a  particular  test  objec¬ 
tive.  A  series  of  test  cases  are  run  within  a  test  cycle. 

•  Test  Incident.  An  unexpected  test  result  requiring  analysis  and  further  investigation. 

•  Test  Log.  A  record  over  time  of  the  details  of  test  execution,  including  events  for  each 
cycle  of  testing.  Typically,  holds  all  activities  for  testing  one  version  of  given  prod¬ 
uct.  Test  results  are  recorded  and  may  reference  a  test  incident 

These  test  objects  can  be  linked  to  test  items,  that  is,  any  software  item  that  is  to  be  tested, 
and  test  tools.  This  allows,  for  example,  identifying  which  testing  tools  are  used  to  run  dif¬ 
ferent  test  cases. 

Incidents  and  problems  identified  in  both  development  activities  and  field  operation 
can  be  recorded.  They  are  maintained  separately.  Problems  may  be  entered  directly  into  the 
problem  database,  or  by  classifying  an  incident  as  a  problem.  Problems  are  given  a  sever¬ 
ity,  resolution  type,  and  status.  They  can  be  assigned  to  a  specific  person  for  resolution  and 
so  support  the  scheduling  of  resources  for  problem  repair.  Documentation  of  actual  reso¬ 
lution  enables  accounting  of  related  costs. 

In  some  cases,  test  log,  incident,  or  problem  data  may  be  generated  by  some  other  tool, 
or  word  processor.  If  this  data  is  in  the  form  of  comma-delimited  ASCII  text  files,  tem¬ 
plates  can  be  created  that  define  how  it  should  be  imported  into  SQA:Manager’s  database. 

Various  types  of  reporting  are  available.  For  a  given  test  cycle,  SQA:Manager  produc¬ 
es  reports  identifying  what  test  cases  were  run  and  their  results,  and  the  summarized  results 
of  all  test  cases.  For  a  given  test  case,  the  results  through  all  test  cycles  can  be  reported. 
Cost  reports  are  available  for  the  cost  of  testing  and  the  cost  of  repair.  General  reporting  on 
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problems  and  incidents  experienced  is  available.  Problem  data  is  also  used  to  generate  re¬ 
liability  metrics  using  Musa’s  logarithmic  Poisson  execution  time  model.  Failure  intensity 
reports  indicate  the  expected  number  of  failures  per  unit  of  time,  whereas  reliability  reports 
give  the  probability  of  failure-free  operation.  Both  types  of  report  provide  additional  infor¬ 
mation  such  as  the  amount  of  additional  testing  time  needed  to  meet  a  targeted  reliability 
objective.  The  accuracy  of  these  estimates  depends  on  the  amount  of  problem  data.  SQA:- 
Manager  documentation  recommends  that  the  reliability  model  is  only  applied  if  there  are 
more  than  100  problem  reports,  testers  are  at  least  25%  through  testing,  and  the  software 
under  test  exceeds  25,000  lines  of  code.  Finally,  reports  on  the  software  under  test,  docu¬ 
ments,  inventory  contents,  and  test  log  are  available. 

Many  of  the  reports  are  available  as  either  tables,  bar  charts,  or  line  graphs.  Before 
printing  any  predefined  report,  the  user  can  change  the  report  title  and  x-  and  y-axis  labels. 
Some  dozen  filters  are  predefined  for  both  incident  and  problems  reports.  These  can  be 
used,  for  example,  to  limit  problem  reporting  to  problems  not  yet  resolved.  The  user  can 
easily  prepare  templates  that  generate  additional  reports,  defining  the  items  to  appear  along 
with  primary  and  secondary  sorting  keys. 

Predefined  templates  are  provided  for  test  plans  and  test  design  specifications.  SQA:- 
Manager  comes  with  three  prepackaged  template  groups,  as  follows: 

•  IEEE  Standards 

a.  IEEE  Std  730,  Software  Quality  Assurance  Plans 

b.  IEEE  Std  829,  Software  Test  Documentation 

c.  IEEE  Std  1008,  Software  Unit  Testing 

d.  IEEE  Std  1028,  Software  Reviews  and  Audits 

•  U.S.  Military  Standards 

a.  MIL-STD-480,  Configuration  Management 

•  U.S.  Department  of  Defense  Data  Item  Descriptions  and  Standards 

a.  DI-MCCR-800 14,  Software  Test  Plans 

b.  DI-MCCR-800 15,  Software  Test  Description 

c.  DI-MCCR-800 17,  Software  Test  Report 

d.  DI-QCIC-80572,  Software  Quality  Program  Plan 

e.  DoD-STD-2167A,  (Excerpt)  Defense  System  Software  Development 

SQArManager  includes  a  set  of  administration  functions.  These  allow  an  administrator 
to  specify  the  relationships  between  a  product,  programs  (a  logical  division  of  products). 
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modules  (a  logical  division  of  programs),  documents,  and  versions.  The  ability  to  reference 
a  program,  or  module,  from  more  than  one  product  supports  software  reuse. 

Security  is  provided  by  identifying  allowable  users,  each  of  whom  is  given  a  password. 
Users  are  distinguished  by  organization  to  allow,  for  example,  identifying  problems  report¬ 
ed  by  customers  as  opposed  to  those  reported  by  testers.  Each  user  is  given  a  particular  set 
of  access  privileges.  In  the  case  of  problem  reporting,  for  example,  a  user  can  be  given  the 
ability  to  add  or  change  descriptions,  set  the  status,  enter  resolution  or  follow-up  action  and, 
for  a  problem,  assign  the  person  responsible  for  resolution.  Setting  up  for  a  new  project  re¬ 
quires  establishing  a  cost  base  that  will  be  used  for  cost  reporting.  Here  the  administrator 
can  define  an  unlimited  number  of  cost  codes,  each  of  which  has  an  associated  hourly  rate. 
Finally,  the  administrator  can  establish  the  necessary  operating  profile,  such  as  the  phone 
numbers  and  communications  ports  to  be  used  with  remote  communications. 

20.2  Observations 

Ease  of  use.  SQA:Manager  provides  a  menu-driven  interface,  where  the  user  may  use 
either  the  keyboard  or  mouse  to  make  selections.  Three  types  of  screens  are  supported:  field 
entry  screens,  check  boxes,  and  item  selector  lists.  After  a  basic  function  has  been  selected, 
the  tool  guides  a  user  through  the  steps  in  that  function,  capturing  information  through  tem¬ 
plates.  Where  appropriate,  a  pick  list  function  is  provided  to  display  the  options  for  a  text 
field  and  allow  the  user  to  select  from  this  list.  Graphical  outputs  are  available  in  the  form 
of  bar  charts,  line  graphs,  and  tables.  Context-sensitive  on-line  help  is  supported,  but  pro¬ 
vides  only  terse  messages. 

A  programming  module  is  provided  to  facilitate  customizing  or  extending  SQA:  Man¬ 
ager.  The  tool  can  be  customized  in  several  ways.  The  text  in  reports  and  help  messages  can 
be  changed,  along  with  field  and  button  labels.  The  contents  of  field  entry  screens  and  item 
selector  lists  can  be  modified.  The  document  templates  can  be  changed,  for  example,  to  re¬ 
flect  particular  organizational  or  product  requirements.  As  previously  mentioned,  the  for¬ 
mat  and  contents  of  reports  can  be  modified,  and  new  report  types  created.  Additionally, 
user-defined  templates  to  convert  comma-delimited  ASCII  text  files  support  importing  of 
problems,  incidents,  and  test  logs. 

Documentation  and  user  support.  The  supplied  documentation  was  well-written  and 
complete.  The  Technical  Reference  Manual  provides  information  to  not  only  customize 
SQA:Manager,  but  to  extend  SQA:Manager  functions  and  integrate  them  with  other  soft- 
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ware  packages.  SQA  staff  were  both  friendly  and  helpful.  They  answered  all  questions 
quickly. 

Problems  encountered.  The  installation  of  SQA:Manager  was  straightforward.  No 
problems  were  encountered  in  its  use. 


20.3  Recent  Changes  and  Planned  Additions 

A  new  version  of  SQA:Manager  running  under  MS-Windows  has  been  released.  A  ver¬ 
sion  for  the  DEC  Ultrix  system  is  expected  to  enter  beta  testing  soon. 


20.4  Sample  Outputs 

Figures  20-1  through  20-18  provide  sample  outputs  from  SQA.Manager. 
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Test  Plan  Report  -  Current  Version 

Plan  IDs  ACTIII02PN 
Plan  Name:  AP  6  AR  TEST  PLAN 

Description:  TEST  PLAN  FOR  VERSION  2.00  OF  ENTIRE 

AP  AND  AR  PROGRAMS  IN  ACT  III  PRODUCT. 

Version:  2.0 

Effective  Date:  11/3/90 

Author:  Beth  J ones 

Location:  C:OCSACT20.PLN 

Referenced  Specifications : 

ID:  ACTIII02DS  Name:  AP  S  AR  DESIGN  SPEC 

Referenced  Software  Components: 

Product 

Name:  ACT  III  2.00 
Program 

Name:  ACCOUNTS  PAYABLE  2.00 
Name:  ACCOUNTS  RECEIVABLE  2.00 

Module 

Name:  CHECK  WRITER  2.00 

Name:  INVOICE  WRITER  1.96 


Test  Plan  Report  -  Revision  History 

Plan  ID:  ACTIII02PN 

Plan  Name:  AP  6  AR  TEST  PLAN 

Description:  TEST  PLAN  FOR  VERSION  2.00  OF  ENTIRE 
AP  AND  AR  PROGRAMS  IN  ACT  III  PRODUCT. 


Version :  1.5 

Effective  Date:  10/8/90 

Author:  Beth  Jones 

Location:  DOCUMENTATION  FILE  CABINET 

Version  Description:  FIRST  VERSION 


Version :  2.0 

Effective  Date:  11/3/90 

Author :  Beth  Jones 

Location :  C : 0CSACT2  0 . PLN 

Version  Description:  CONTAINS  NEW  SECTIONS  FOR  TESTING 

PRINTING  OF  CHECKS  AND  INVOICES 


Figure  20-1.  SQA:Manager  Test  Plan  for  ACTIII02PN 
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Test  Specification  Report  -  Current 

Test  Specification  ID:  ACTIII02DS 
Test  Specification  Name:  AP  &  AR  DESIGN  SPEC 
Description:  TEST  DESIGN  SPEC  FOR  VERSION  2.00 
AP  AND  AR  PROGRAMS  IN  ACT  III 

Version:  1.56 

Effective  Date:  12/6/90 

Author:  Beth  Jones 

Location:  C : OCSACT156 . DSN 

Version  Description:  SAME  AS  VER  1.50  PLUS  FIXES  FOR 

COSMETIC  PROBLEMS. 

Referenced  Procedures: 


ID:  CHKRUNS 

Name: 

CHECK  GENERATION  TESTS 

ID:  INVRUNS 

Name: 

INVOICE  GENERATION 

Refereuced  Test  Cases: 

ID:  CHKDATA 

Name: 

CHECKING  DATA  PREP  TESTS 

ID:  CHKPOST 

Name: 

CHECK  POSTING  TESTS 

ID:  CHKPRN 

Name: 

CHECK  PRINTING  TESTS 

ID:  CHKRPT 

Name: 

CHECK  REPORT  TESTS 

ID:  CHKSEL 

Name: 

CHECK  SELECTION  TESTS 

ID:  CHKSRT 

Name: 

CHECK  SORTING  TESTS 

ID:  INV0001 

Name: 

OPEN  INVOICE  REPORT  TEST 

ID:  INVPRN 

Name: 

INVOICE  PRINTING  TESTS 

Referenced  Software 

Components : 

Program 

Name:  ACCOUNTS  PAYABLE  2.00 
Name:  ACCOUNTS  RECEIVABLE  2.00 


Module 

Name:  CHECK  WRITER  2.00 

Name:  INVOICE  WRITER  1.95 


Test  Specification  Report  -  Revisions 

Test  Specification  ID:  ACTIXI02DS 
Test  Specification  Name:  AP  C  AR  DESIGN  SPEC 
Description:  TEST  DESIGN  SPEC  FOR  VERSION  2.00 
AP  AND  AR  PROGRAMS  IN  ACT  III 


Version:  1.56 

Effective  Date:  12/6/90 

Author :  Beth  Jones 

Location:  C:OCSACT156.DSN 

Version  Description:  SAME  AS  VER  1.50  PLUS  FIXES  FOR 

COSMETIC  PROBLEMS. 


Figure  20-2.  SQA.Manager  Test  Specification  Report  for  Test  Spec  ACTIII02DS 
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Test  Case  Report  -  Current 
Test  Case  ID  INVPRN 

Test  case  Name:  INVOICE  PRINTING  TESTS 
Description;  TESTS  FOR  PRINTING  INVOICES. 

Tool  ID: 

Requirement  ID: 

Version : 

Effective  Date: 

Developer : 

Specification  Location: 

Version  Description:  2ND  VER 
C : CASESNDATAINVPRN . IN 
C : CASESRESULTSNVPRN . OUT 

Referenced  Software  Components: 

Program 

Name:  ACCOUNT  PAYABLE  2.00 
Module 

Name:  INVOICE  WRITER  1.95 


C : CASE SPEC SNVPRN . DOC 
-  ADD  NEW  PAGE  LENGTH  ROUTINE 


1.2 

12/5/90 


Test  Case  Report  -  Revisions 
Test  Case  ID  INVPRN 

Test  case  Name:  INVOICE  PRINTING  TESTS 
Description:  TESTS  FOR  PRINTING  INVOICES. 

Tool  ID: 

Requirement  ID: 


Version :  1 . 0  9 

Effective  Date:  9/10/90 

Developer :  Mike  Brown 

Specif ieation  Location:  C : CASESPECSNVPRN . DOC 

Version  Description:  FIRST  VERSION 
C : CASESNDATAINVPRN . IN 

C : CASESRESULTSNVPRN . OUT  M 


Version :  1.2 

Effective  Date:  12/5/90 

Developer : 

Specification  Location;  C; CASESPECSNVPRN. DOC 

version  Description:  2ND  VER  -  ADD  NEW  PAGE  LENGTH  ROUTINE 
C : CASESNDATAINVPRN . IN 
C : CASESRESULTSNVPRN . OUT 

Figure  20-3.  SQArManager  Test  Case  Report  for  Test  Case  INVPRN 
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rest  Procedure  Report  -  Current 
lest  Procedure  ID  CHKRUNS 

Test  Procedure  Names  CHECK  GENERATION  TESTS 

Description:  TESTS  PROM  ENTERING  CHECK  DATA  TO 
POSTING  CHECKS . 

Version :  2.0 

Effective  Date:  10/30/90 

Developer :  Beth  Jones 

Proaedure  Location: 

Version  Description:  MAJOR  REV  -  ADDS  PROCEDURES  FOR  CHECK 

POSTING  AND  SELECTING  TESTS. 

Referenced  Software  Components : 

Program 

Name:  ACCOUNTS  PAYABLE  2.00 
Module 

Name:  CHECK  WRITER  2.00 


Test  Procedure  Report  -  Revisions 
Test  Procedure  ID  CHKRUNS 

Test  Procedure  Name:  CHECK  GENERATION  TESTS 

Description:  TESTS  FROM  ENTERING  CHECK  DATA  TO 
POSTING  CHECKS. 

Version s  1.0 

Effective  Date:  1/3/90 

Developer:  Beth  Jones 

Procedure  Location: 

Version  Description:  FIRST  VERSION 

Version :  57o 

Effective  Date:  10/3090 

Developer :  Beth  Jones 

Procedure  Location: 

Version  Description:  MAJOR  REV  -  ADDS  PROCEDURE  FOR  CHECK 

POSTING  AND  SELECTING  TESTS. 


Figure  20-4.  SQA:Manager  Test  Procedure  Report  for  Procedure  CHKRUNS 
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Software  I teas  Report 
Date:  6/7/92 

Product:  ACT  III  2.00 

Program:  ACCOUNTS  PAYABLE  2.00 

Module:  DATABASE  MANAGER  1.80 

Module:  CHECK  WRITER  2.00  f 

Module:  JOURNAL  2.00 
Module:  USER  INTERFACE  1.80 
Program:  ACCOUNTS  RECEIVABLE  2.00 
Module:  DATABASE  MANAGER  1.80 
Module:  INVOICE  WRITER  1.95 
Module:  JOURNAL  2.00 

Module:  USER  INTERFACE  1.80  # 

Program:  GENERAL  LEDGER  2.00 

Module:  DATABASE  MANAGER  1.80 
Module:  JOURNAL  2.00 
Module:  USER  INTERFACE  1.80 


Figure  20-5.  SQA:Manager  Software  Items  Report 


Test  Tool 

Report  -  List 

Tool  ID 

Tool  Name 

AUTOT 

AutoTester 

ROBOT 

SQA -.Robot  OS/2 

ROBOT  W 

SQA: Robot  Windows 

SQAM 

SQAManager 

TESTPRO 

TestPro 

Test  Tool  Report 

Tool  ID: 

TESTPRO 

Tool  Name: 

TestPro 

version: 

3.1 

Vendor : 

Sterling  Software 

Purpose : 

Capture/Playback  for  MS-DOS 

Date  Acquired: 
Location : 

6/15/91 

Figure  20-6.  SQArManager  Test  Tool  Report 
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Test  Log  Report 

Test  Log  Name:  ACT  III  VER  2.00  TESTING  Test  Log  ID:  ACT3320 
Ref'd.  Test  Procedure: 

Description:  Testing  for  Alpha  Releases  A-H 
of  ACT  III  Ver  2.0 


Test  Cycle 

Date 

Time 

Tester 

Test  Case  ID  Incident 

A 

12/26/90 

08:00 

MIKEB 

CHKSRT 

1 

Garbage  in  upper  left 

corner . 

A 

12/26/90 

08:30 

MIKEB 

CHKPRN 

2 

Truncated  Payee  Naaes 

(over  40 

chars) 

A 

12/26/90 

09:00 

MIKEB 

CHKRPT 

3 

Memory  problem. 

A 

12/26/90 

10:00 

MIKEB 

4 

Error  202  when  AP  selected  from  Main  Menu  -  intermittent 

A 

12/26/90 

11:15 

MIKEB 

5 

Typo 

A 

12/26/90 

15:00 

MIKEB 

CHKSRT 

6 

Checks  don't  appear  in  descending  orderF 

A 

12/26/90 

16:30 

MIKEB 

CHKSRT 

0 

Wrong  system  setup  -tests  aborted 

A 

12/27/90 

08:00 

MIKEB 

CHKSRT 

7 

A 

12/27/90 

08:00 

MIKEB 

CHKSRT 

0 

Second  run  was  OK  -  operator  error  (I  think) 

A 

12/27/90 

10:00 

MIKEB 

CHKDATA 

10 

A 

12/27/90 

11:00 

MIKEB 

CHKDATA 

0 

A 

12/27/90 

13:01 

MIKEB 

9 

404  Error 

A 

12/27/90 

13:30 

MIKEB 

INV0001 

0 

A 

12/27/90 

14:00 

MIKEB 

XNV0001 

11 

A 

12/27/90 

16:00 

MIKEB 

12 

A 

12/27/90 

16:05 

MIKEB 

CHKDATA 

14 

A 

12/27/90 

17:00 

MIKEB 

CHKDATA 

0 

A 

12/27/90 

17:15 

MIKEB 

0 

A 

12/27/90 

17:50 

MIKEB 

17 

A 

12/27/90 

18:00 

MIKEB 

18 

A 

12/27/90 

18:30 

MIKEB 

20 

A 

12/28/90 

08:00 

MIKEB 

XNV0001 

23 

A 

12/28/90 

09:00 

MIKEB 

INV0001 

0 

A 

12/28/90 

19:00 

MIKEB 

21 

B 

12/28/90 

10:00 

MIKEB 

CHKSRT 

0 

B 

12/29/90 

08:30 

MIKEB 

CHKPRN 

0 

B 

12/29/90 

11:15 

MIKEB 

INVPRN 

0 

B 

12/29/90 

13:00 

MIKEB 

CHKDATA 

0 

Figure  20-7.  SQA:Manager  Test  Log  Report 
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Test  Log  Nana:  ACT  III  VER  2.00  TESTING  Test  Log  ID:  ACT3320 
Ref'd.  Test  Procedure: 

Description:  Testing  for  Alpha  Releases  A-H 


Passed 
Failed 
Aborted 
Incidents 
Elapsed  Tine 


of  ACT  III  Ver  2.0 
9 
9 
1 

17 

21.30 


Figure  20-8.  SQA:Manager  Test  Case  Report  for  Test  Case  INVPRN 


Problems  By  Repair  Person 

Eat  Fix  Problem 
Date  ID  Program  Name 


Asga'd 

Dev. 


Short  Deac. 


3 

5 

6 
8 

10 

11 

12 

13 

14 


ACCOUNTS  PAYABLE:  2.00 
ACCOUNTS  PAYABLE:  2.00 
ACCOUNTS  PAYABLE:  2.00 
GENERAL  LEDGER:  2.00 
ACCOUNTS  RECEIVABLE .- 
ACCOUNTS  RECEIVABLE: 
GENERAL  LEDGER:  2.00 
GENERAL  LEDGER:  2.00 
ACCOUNTS  RECEIVABLE: 


2.00 

2.00 


2.00 


15 

GENERAL  LEDGER:  2 . 

.00 

CATHYW 

1/8/91 

2 

ACCOUNTS 

PAYABLE; 

2.00 

CATHIW 

1/8/91 

7 

ACCOUNTS 

PAYABLE: 

2.00 

CATHYW 

1/9/91 

9 

ACCOUNTS 

PAYABLE: 

2.00 

CATHYW 

1/15/91 

1 

ACCOUNTS 

PAYABLE: 

2.00 

CATHYW 

1/15/91 

4 

ACCOUNTS 

RECEIVABLE: 2 . 00 

CATHYW 

Chechaorting-  wrong  order 
Checkprinting-  Can't  use  ind  checks 
Control  Account-  selection  error 
Help  doesn't  work 

Open  Invoice  Rpt-  last  one  miasing 


Post invoices-  missing  posting  date 
GL-  won't  accept  Ml/DD/YY  format 
Checkvriter-  memory  error. 
Checkrecording-  Tasks  Menu  -  in  loop 
Chart  of  Accounts-  Corrupted 
Checkwriter-  garbage 
Checkdata-  Unable  to  execute  line  1232 


Figure  20-9.  SQA:Manager  Problems  Table 
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Fixed  Problems  Read/  for  Retest 


Problem 

Submitter's 

ID 

ID 

Status 

Fixed  Program 

Module/Subsystem  Short 

1 

MIKZB 

Submitted 

Check 

2 

MIKEB 

Assigned 

ACCOUNTS  PAYABLE: 

2.50 

CHECK  WRITER:  2.00  Check 

3 

MIKZB 

Submitted 

Check 

4 

MIKEB 

Submitted 

Check 

5 

MIKEB 

Submitted 

Check 

6 

MIKEB 

Submitted 

Contr 

7 

MIKEB 

Assigned 

ACCOUNTS  PAYABLE: 

2.50 

USER  INTERFACE:  1.80  Check 

8 

MIKEB 

Submitted 

Help 

9 

MIXEB 

Assigned 

ACCOUNTS  PAYABLE: 

2.50 

DATABASE  WRITER:  1.80  Chart 

10 

MIKEB 

Submitted 

11 

MIKEB 

Submitted 

Open 

12 

MIKEB 

Submitted 

13 

MIKEB 

Submitted 

14 

MIKEB 

Submitted 

Posti 

15 

MIKEB 

Assigned 

ACCOUNTS  PAYABLE: 

2.50 

GL  - 

Figure 

20-10.  SQA:Manager  Fixed  Problems  Ready  for  ReTest 

Repair  Cost 

Program  Name:  ACCOUNTS  PAYABLE 

Version i  2.00 


Cost  of  Repair 
Date  Problem  ID 


12/28/90  1 

12/28/90  2 

12/28/90  3 

12/28/90  5 

12/28/90  6 

12/28/90  7 

12/28/90  9 

Total 


Developer  ID 


CATHYW 

CATHYW 


CATHYW 

CATHYW 


Time  (hours)  Cost 


0.00 

4.00  84.00 


1.00  21.00 

2.00  42.00 

147.00 


Figure  20-11.  SQA:Manager  Cost  of  Repair  Table 
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Test  Log  Report 

Test  Log  Name:  ACT  III  VER  2.00  TESTING  Test  Log  ID:  ACT3320 
Ref'd  Test  Procedure: 

Description :  Testing  for  Alpha  Releases  A-H 
of  ACT  III  Ver  2.0 


Cost  of  Testing 


Date 

Tine 

Tester 

Tine  (hours) 

Cost 

12/26/90 

08:00 

MIKEB 

0.50  - 

10.50 

12/26/90 

08:30 

MIKEB 

1.00 

21.00 

12/26/90 

09:00 

MIKEB 

0.75 

15.75 

12/26/90 

10:00 

MIKEB 

0.50 

10.50 

12/26/90 

11:15 

MIKEB 

0.25 

5.25 

12/26/90 

15:00 

MIKEB 

1.50 

31.50 

12/26/90 

16:30 

MIKEB 

0.50 

10.50 

12/27/90 

08:00 

MIKEB 

0.25 

5.25 

12/27/90 

10:00 

MIKEB 

0.50 

10.50 

12/27/90 

11:00 

MIKEB 

2.00 

42.00 

12/27/90 

13:01 

MIKEB 

0.75 

15.75 

12/27/90 

13:30 

MIKEB 

0.50 

10.50 

12/27/90 

14:00 

MIKEB 

0.75 

15.75 

12/27/90 

16:00 

MIKEB 

0.75 

15.75 

12/27/90 

16:05 

MIKEB 

0.25 

5.25 

12/27/90 

17:00 

MIKEB 

1.00 

21.00 

12/27/90 

17:15 

MIKEB 

1.00 

21.00 

12/27/90 

17:50 

MIKEB 

0.50 

10.50 

12/27/90 

18:00 

MIKEB 

0.25 

5.25 

12/27/90 

18:30 

MIKEB 

0.25 

5.25 

12/28/90 

08:00 

MIKEB 

3.00 

63.00 

12/28/90 

09:00 

MIKEB 

0.75 

15.75 

12/28/90 

10:00 

MIKEB 

0.50 

10.50 

12/28/90 

19:00 

MIKEB 

1.50 

31.50 

12/29/90 

08:30 

MIKEB 

0.50 

10.50 

12/29/90 

11:15 

MIKEB 

0.30 

6.30 

12/29/90 

Total 

13:00 

MIKEB 

0.75 

15.75 

442.05 

Figure  20*12.  SQA:Manager  Cost  of  Testing 
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Reliability  Analysis 
Poisson  Geometric  model 


Date:  06/07/92 

Program  Name :  TESTPROGRAM 

version:  1.00A 


Total  Testing  Time  (CPU  hrs): 
Total  Failures  Reported: 
Target  Reliability: 

Confidence  Interval: 


0 

137 


0.80  /1.0  CPU  hours 
90% 


Present  Reliability 
Low  Limit: 
Most  Likely 
High  Limit: 


0.00  /I . 0  CPU  hours 
0.00  /1.0  CPU  hours 
0.00  /10  CPU  hours 


Additional  Time  Needed 

Low  Limit: 

Most  Likely: 
High  Limit: 


for  Execution 

CPU  Time  (hrs)  Calendar  Time  (days) 
224.8  936.9 

276.1  1150.4 

357.5  1489.8 


Additional  Failures  to 
Low  Limit: 

Most  Likely: 
High  Limit: 


be  Found 
340  failures 
428  failures 
568  failures 


Figure  20-15.  SQA:Manager  Reliability  Analysis  Table  and  Graph 
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Failure  Intensity  Analysis 
Poisson  Geonetric  model 

Date:  06/07/92 

Program  Nane  TESTPROGRAM 

Version:  1.00A 

Total  Testing  Time  (CPU  hrs):  0 
Total  Failures  Reported:  137 

Target  Failure  Intensity:  0.008000  failures/CPU  sec 

Confidence  Interval:  90% 


Initial  Failure  Intensity 


Lav  Limit: 

0.422475 

failures/CPU 

sec 

Most  Likely: 

0.588194 

failures/CPU 

fee 

High  Limit: 

0.832898 

failures/CPU 

sec 

Present  Failure  Intensity 

Low  Limit : 

0.075974 

failures/CPU 

sec 

Most  Likely .- 

0.063783 

failures/CPU 

sec 

High  Limit: 

0.054462 

failures/CPU 

sec 

Additional  Time  Needed 

for  Execution 

CPU  Time  (hrs) 

Calendar  Time  (days) 

Low  Limit: 

1.5 

6.2 

Most  Likely: 

1.9 

7.8 

High  Limit: 

2.5 

10.3 

Additional  Failures  to 

be  Found 

Low  Limit: 

96  failures 

Most  Likely :  128  failures 

High  Limit:  180  failures 


Fai lure  I ntens i 1 y 
For  ACT  111  Version  2.  00 


Figure  20-16.  SQA:Manager  Failure  Intensity  Table  and  Graph 
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Incidents  by  Symptom 
For  ACT  III  Version  2.00 


Figure  20-17.  SQA:Manager  Plot  of  Incidents  by  Symptom 

Problems  By  Severity 
For  ACT  I  i  1  Ver  s  ion  2.  00 


Figure  20-18.  SQA:Manager  Plot  of  Problems  by  Severity 
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21.  SRE  TOOLKIT 

The  SRE  Toolkit  supports  reliability  growth  modeling  using  the  Musa-Okumoto  soft¬ 
ware  reliability  models.  It  takes  a  record  of  failure  events,  in  terms  of  elapsed  execution 
time  from  the  start  of  test,  and  estimates  various  software  reliability  measures  that  track  the 
progress  of  testing.  This  is  particularly  useful  during  system  testing  when  the  underlying 
faults  causing  failures  are  removed  and  hence  “reliability  growth”  is  occurring. 


21.1  Tool  Overview 

The  SRE  Toolkit  is  available  from  Software  Quality  Engineering  who  hold  a  license 
from  AT&T  for  this  version  of  the  AT&T  software  reliability  engineering  tools  and  accom¬ 
panying  training.  The  tools  are  provided  as  part  of  a  consulting  and  training  package.  This 
package  typically  includes  needs  assessment  and  planning.  Assistance  in  conducting  and 
evaluating  a  pilot  application  is  also  available.  The  three  day  training  courses  are  held  at  a 
public  location  or  at  a  client’s  site.  Attendance  at  a  public  course  currently  costs  $995  per 
person,  while  up  to  20  people  can  attend  an  on-site  course  for  a  total  cost  of  $10,000. 

The  tool  has  been  available  since  1990.  There  are  two  versions.  One  runs  under  Unix 
System  V  on  any  supporting  hardware.  The  other  runs  under  DOS  (release  3.3  onwards)  on 
an  AT&T  PC  or  compatible.  Both  versions  of  the  toolkit  require  the  awk  program.  While 
screen  graphics  are  available  under  MS-DOS,  the  Unix  version  of  the  tools  generate  pic 
commands  for  a  graphics  output  device  and,  therefore,  need  the  associated  pic  utilities.  The 
evaluation  was  performed  on  the  Unix  version  3. 1 1  of  the  toolkit 

The  two  main  tools  in  the  toolkit  are  est,  the  reliability  estimation  tool,  and  plot ,  the 
graphics  support  tool.  For  est,  the  user  specifies  whether  an  exponential  or  logarithmic  re¬ 
liability  model  is  required  and  the  failure  intensity  objective  that  will  be  used  to  determine 
when  to  stop  testing.  He  can  specify  whether  failure  data  entries  should  be  interpreted  to 
correspond  to  individual  failure  events,  or  to  the  number  of  failures  that  occurred  since  the 
previous  failure  entry  or  start  of  test  interval  (grouped  data  analysis).  A  testing  compression 
factor  specifies  the  desired  ratio  of  field  execution  time  to  test  execution  time  allowing,  for 
example,  more  stress  to  be  placed  on  in-house  testing  than  field  testing.  In  the  case  of  the 
exponential  model,  the  user  can  also  specify  a  failure  time  adjustment  to  adjust  failure  times 
to  take  into  account  the  incremental  delivery  of  software  to  system  test.  For  the  logarithmic 
model,  a  failure  intensity  decay  parameter  determines  the  rate  of  exponential  decay. 
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The  est  tool  uses  the  fitted  model  to  estimate  several  reliability  measures  over  a  range 
of  confidence  limits.  These  reliability  measures  include  the  present  failure  intensity  and  the 
additional  failures,  test  execution  time,  and  work  days  required  to  meet  the  specified  failure 
intensity  objective.  In  the  case  of  an  exponential  reliability  model,  the  total  number  of  fail¬ 
ures  that  would  be  experienced  after  an  infinite  amount  of  execution  time  is  also  given.  Fi¬ 
nally,  the  expected  calendar  date  when  the  failure  intensity  objective  will  be  met  and  the 
additional  calendar  time  needed  for  testing  that  this  implies  are  reported.  In  addition  to  such 
tabular  data,  est  generates  a  file  of  plot  commands  for  graphic  output. 

Prototype  tools  are  included  in  the  toolkit: 

•  resrusg.  Uses  simple  regression  analysis  to  estimate  the  testing  resource  usage  param¬ 
eters.  It  produces  summary  statistics  for  each  recording  period  and  estimates  showing 
the  resources  consumed  per  unit  execution  time  and  resources  consumed  per  failure 
parameter,  resrusg  also  generates  plots  that  show  how  well  the  regressions  tit  the  orig¬ 
inal  data. 

•  reldem.  Plots  reliability  demonstration  charts  that  indicate,  for  a  given  failure  inten¬ 
sity  objective,  whether  testing  should  continue. 

•  predat.  Plots  the  completion  date  for  testing,  that  is,  when  the  failure  intensity  objec¬ 
tive  is  met  versus  failure  intensity  objective.  It  also  produces  a  table  indicating  testing 
periods  where  a  particular  testing  resource  is  a  limiting  resource. 

«  minfio.  Plots  the  total  life  cycle  costs  versus  failure  intensity  objective.  It  includes  the 
system  test  and  operational  life  cost 

•  logmod.  Plots  the  logarithmic  reliability  growth  model  using  model  parameters,  ini¬ 
tial  failure  intensity,  and  the  failure  intensity  decay  factor. 

•  expmod.  Plots  the  exponential  reliability  growth  model  using  model  parameters,  ini¬ 
tial  failure  intensity,  and  total  failures  after  infinite  execution  time. 


21,2  Observations 

Ease  of  use.  The  tools  are  written  as  shell  scripts  (or  batch  files  in  the  case  of  the  MS- 
DOS  version).  Their  operation  can  be  tailored  using  parameters  that  are  given  in  a  param¬ 
eter  file  or,  in  some  cases,  included  with  a  tool’s  input  data.  The  parameters  range  from 
specifying  the  formatting  of  an  output  plot  to  such  details  as  the  cost  per  failure  identifica¬ 
tion  resource  hour.  They  provide  considerable  flexibility  in  the  application  of  each  tool. 

Documentation.  The  documentation  provided  in  the  form  of  Unix-like  man  pages  is 
both  helpful  and  extensive.  The  definition  of  the  file  format  for  failure  data  facilitates  im¬ 
porting  data.  Installation  was  straightforward  although  minor  problems  arose  due  to  system 
dependencies. 
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Problems  encountered.  The  tools  operated  as  described  in  the  documentation. 

21.3  Sample  Outputs 

Figures  21-1  through  21-9  provide  sample  outputs  from  SRE  Toolkit. 
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FAILURE  PARAMETER  FILE  IS  t*t_*tg. fp 
FAILURE  TIME  ANALYSIS  HILL  BE  DONE 

TEST  COMPRESSION  FACTOR  OF  IS. I  HILL  BE  APPLIED  DURING  ANALYSIS 

FAILURE  TIME  FILE  IS  t«t_Stg.ft 

FAILURE  TIMES  HERE  ADJUSTED 

ADJUSTED  FAILURE  TIMES  FILE  IS  . »d 

GENERATING  OUTPUT  REPORT 


SOFTWARE  RELIABILITY  ESTIMATION 
EXPONENTIAL  (BASIC)  MODEL 
TST  DATA  SET  WITH  STAGED  DEVELOPMENT  ADJUSTMENTS 


BASED  ON  SAMPLE  OF 
TEST  EXECUTION  TIME  IS 
FAILURE  INTENSITY  OBJECTIVE 
CURRENT  DATE  IN  TEST 
TIME  FROM  START  OF  TEST  IS 


136  TEST  FAILURES 
31.4559  CPU— HR 

IS  3.4  FAILURES/1 0 0 0-CPU-HR 
861109 
96  DAYS 


CONF. 

LIMITS 

MOST 

CONF. 

LIMITS 

95* 

908 

75% 

50% 

LIKELY 

50% 

75% 

90% 

95% 

TOTAL  FAILURES 

138 

139 

139 

140 

141 

143 

144 

145 

147 

*  FAILURE  INTENSITIES 

(FAXLURES/1000— CPU— HR) 

**••* 

INITIAL 

1190 

1334 

1303 

1370 

1467 

1566 

1636 

1710 

1757 

PRESENT 

38.59 

31.35 

36.19 

41.49 

50.30 

60.54 

68.95 

78.78 

85.67 

•*«  ADDITIONAL  REQUIR1MENTS  TO  MEET  FAILURE  INTENSITY 

OBJECTIVE  **« 

FAILURES 

3 

3 

3 

3 

5 

6 

7 

9 

10 

TEST  EXEC.  TIME 

13.91 

13.79 

15.38 

16.84 

19.33 

33.30 

34.53 

37.33 

39.15 

WORE  DAYS 

3.05 

3.39 

3.70 

4.15 

4.95 

5.97 

6.87 

8.00 

8.85 

COMPLETION  DATE 

861113 

•61113 

•61113 

•61114 

861114 

•61117  861118 

861119 

861120 

GENERATING  PLOT  COMMANDS.  COMPLETED!  PLOT  COMiAND  FILE  IS  t«t_«tg . pc . 


Figure  21*1.  SRE  TooHclt  Generated  Reliability  Measures 


<5  9  -134  18  224 

TST  DATA  SET  WITH  STAGED  DEVELOPMENT  ADJUSTMENTS 

Figure  21*2.  SRE  Toolkit  Failure  vs.  Execution  Time  Plot 
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Initial  Intensity  VS  Exec.  Time 


TST  DATA  SET  WITH  STAGED  DEVELOPMENT  ADJUSTMENTS 


Figure  21-3.  SRE  Toolkit  Initial  Intensity  vs.  Execution  Time  Plot 
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Period  Ident .  Computer  Appr.  Corr.  Items 


Rate 

Rate 

Fail.Int. 

Work 

Corr. 

1 

26.8949 

76.5281 

39.1198 

50.6 

7 

2 

15.4162 

25.7965 

13.3607 

87.5 

6 

3 

22.3404 

29.7872 

27.6596 

59 

5 

4 

16.3866 

31.9328 

10.9244 

170 

19 

5 

24.2537 

16.4179 

10.4478 

103.1 

19 

6 

12.6506 

11.8976 

5.42169 

66 

19 

7 

6.17978 

7.30337 

3.37079 

92.7 

10 

8 

6.39098 

9.58647 

4.51128 

105.5 

12 

9 

4.90716 

8.32891 

3.97878 

75.1 

20 

10 

5.13158 

6.28947 

1.31579 

105.1 

19 

Failure  Identification  Resources 

thetai  -  7.19208  hrs/CPU-hr,  mui  -  0.571399  hrs/failure 

Failure  Correction  Resources 
muf  -  6.05505  hr s/fix 
Computing  Resources 

thetac  -  2.93493  hrs/CPU-hr,  muc  «  1.6195  hrs/failure 
Reformatted  input  is  in  file  ex6b.out 


Work  Ttnus  Itcnu  Corrected 


Falhm  OrntmUsm  Rssseras 


Figure  21-6.  SRE  Toolkit  Testing  Resource  Usage  Parameter  Estimation 
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LIMITING  RESOURCE  PERIODS 
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Figure  21*7.  SRE  Toolkit  Reliability  Demonstration  Chart 
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Figure  21  -9.  SRE  Toolkit  Life  Cycle  Cosi 
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22.  T 

T  generates  test  data  from  requirements  information  and  automatically  provides  tracing 
between  tests  and  defined  software  actions.  Its  goal  is  to  generate  the  minimum  number  of 
traceable,  unique  test  cases  that  will  exercise  every  operation  and  each  of  a  set  of  vendor- 
defined  probable  errors  at  least  once.  Test  adequacy  is  assessed  based  on  requirements  cov¬ 
erage,  input  domain  coverage,  output  range  coverage  and,  optionally,  structural  coverage. 
T  can  be  used  during  any  software  development  stage;  during  maintenance  test  data  is  gen¬ 
erated  for  software  changes  only.  T  is  already  used  by  various  government  organizations, 
including  the  Naval  Avionics  Center,  the  Jet  Propulsion  Laboratories,  Naval  Coastal  Sys¬ 
tems  Center,  and  U.S.  Army  Forts  Monmouth  and  Sill. 

Runner  is  a  companion  tool  that  provides  test  capture/playback  for  C. 


22.1  Tool  Overview 

T  was  developed  by  Programming  Environment  Inc.  In  addition  to  T  and  Runner,  this 
organization  markets  consultancy  and  training  services,  and  supports  tool  users  with  a  quar¬ 
terly  newsletter.  T  has  been  available  since  1987  and  has  over  1,000  users.  It  is  available 
for  PC/MS-DOS  and  VAX/VMS  platforms,  and  various  workstations  under  Unix.  Training 
is  a  prerequisite  for  tool  purchase  and  costs  $1,500  per  person  at  Programming  Environ¬ 
ments,  Inc.  or  $10,000  for  an  on-site  workshop.  At  the  time  of  evaluation,  prices  for  T  itself 
started  at  $7,000.  T,  and  training  materials,  are,  however,  provided  free  to  any  university 
that  teaches  courses  on  software  testing. 

Interfaces  between  T  and  some  leading  CASE  tools  are  available.  An  interface  to  Team¬ 
work  is  marketed  by  Cadre  Technologies,  Inc.  and  Interactive  Development  Environments 
markets  an  interface  to  StP.  (Boeing  has  developed  a  proprietary  interface  for  the  Exceler- 
ator  CASE  and  Texas  Instruments  for  their  Information  Engineering  Facility.)  Support  for 
reverse  engineering  of  TSDL  specifications  from  existing  code  is  also  available.  In  this 
case.  Cadre  markets  tools  to  reverse  generate  a  substantial  part  of  TSDL  specifications  from 
Ada,  C,  or  Fortran.  (Boeing  is  developing  a  proprietary  tool  providing  the  same  function 
for  COBOL.  Boeing  is  also  developing  tools  to  generate  Ada  and  C  code  from  TSDL  spec¬ 
ifications.)  Code  generation  may  also  be  supported  by  IBM’s  current  effort  to  convert  from 
TSDL  to  Z. 
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To  support  test  execution,  interfaces  between  T  and  the  AutoTester  Corp.  product  Au- 
toTester,  the  Mercury  Interactive  Corp.  product  XRunner,  and  Tiburon’s  Ferret  capture/ 
playback  tools  are  commercially  available. 

The  examination  was  performed  on  T  version  B3.0,  running  on  a  Sun  SPARC  worksta¬ 
tion  under  Unix.  This  version  was  a  demonstration  copy  of  T,  fully  functional  with  the  ex¬ 
ception  that  only  a  limited  number  of  data  definitions  and  sentences  can  be  processed  at  one 
time  and  some  reports  are  not  generated.  Additionally,  the  full  set  of  test  design  rules  is  pro¬ 
prietary  information  only  available  with  purchase  of  full  version;  as  a  consequence,  the 
sampling  rules  cannot  be  changed  and  features  such  as  random  sampling  are  not  available 
in  the  demonstration  version. 

Before  using  T,  the  user  must  prepare  a  description  of  software  actions,  data,  events, 
and  condition  states.  At  the  system  level,  this  type  of  requirements  information  may  be  de¬ 
rived  from  sources  such  as  data  flow  diagrams,  state  transition  diagrams,  entity  relationship 
diagrams,  and  control  diagrams.  At  the  design  or  unit  level,  module  descriptions  will  be  the 
primary  source.  The  description  is  specified  in  the  T  Software  Description  Language 
(TSDL),  a  superset  of  the  Semantic  Transfer  Language  (STL)  [IEEE  1992].  TSDL  allows 
specification  of  operation  statements  and  definitions  for  data  items,  conditions,  and  states. 
It  also  supports  event  types  with  multiple  inheritance  for  operations  supported  by  temporal 
conditions.  Standard  templates  are  provided  to  help  write  TSDL  descriptions.  The  collec¬ 
tion  of  resulting  ASCII  files  is  called  a  Software  Description  File  (SDF). 

Once  an  SDF  has  been  prepared,  the  first  component  of  T,  called  Tverify,  can  be  run. 
This  translates  the  SDF  into  a  Software  Description  Data  Base  ( SDDB )  and  then  checks  this 
database  for  syntax  errors  and  to  see  whether  it  contains  the  necessary  and  sufficient  infor¬ 
mation  for  test  case  design  and  preparation.  This  verification  also  provides  metrics  such  as 
counts  of  actions,  states,  conditions,  events,  and  data  items  defined  in  the  SDF,  and  error 
counts.  A  cross-reference  report  shows  where  every  item  is  used  (this  report  is  not  available 
with  the  demonstration  version). 

After  successful  verification,  the  user  can  request  T  to  design  test  cases.  Here  the  sec¬ 
ond  T  component,  called  Tdesign,  takes  information  from  the  SDF  to  group  actions  into  ap¬ 
propriate  states.  Test  design  rules  are  applied  to  partition  data  item  domains  such  that  all  of 
the  values  of  the  data  item  in  a  partition  will  probably  be  processed  in  the  same  manner. 
The  test  design  rules  used  by  T  are  derived  from  the  following  test  techniques: 
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•  Functional  testing.  The  input  domain  is  partitioned  into  classes  such  that  each  mem¬ 
ber  of  a  class  causes  a  given  action  to  be  executed.  Test  data  is  selected  from  these 
partitions  that  will  cause  all  actions  to  be  exercised. 

•  Equivalence  class  partitioning.  The  input  domain  is  partitioned  into  a  number  of 
equivalence  classes  such  that  a  test  of  a  representative  value  of  each  class  is  equiva¬ 
lent  to  a  test  of  any  other  value.  The  minimum  set  of  test  data  that  will  invoke  as  many 
different  input  conditions  as  possible  is  selected  from  these  partitions. 

•  Boundary  value  analysis.  The  input  domain  is  partitioned  into  a  number  of  equiva¬ 
lence  classes.  Test  data  is  chosen  that  lies  at  the  edge  of  these  partitions  to  reflect  the 
boundaries  of  the  input  domain. 

•  Cause-effect  graphing.  The  input  and  output  domains  are  partitioned  into  classes  that 
specify  which  input  classes  cause  which  outputs  to  occur.  Test  data  is  selected  that 
will  cause  all  effects  to  be  exercised. 

•  State-directed  testing.  Test  data  is  selected  to  cause  every  transition  between  states  to 
be  exercised. 

•  Event-directed  testing.  Test  data  is  selected  to  cause  every  event  (some  signal  passing 
from  the  external  world)  to  be  exercised. 

A  fault-directed  approach  is  used  to  select  samples  from  the  partitions  yielded  by  the 
above  techniques.  This  approach  guides  the  selection  of  both  valid  and  invalid  samples  that 
are  likely  to  detect  errors.  By  default  T  selects  the  following  samples: 

•  For  the  valid  subdomain: 

a.  Low  boundary  value, 

b.  Just  above  low  boundary  value, 

c.  Reference  value  (midway  between  the  low  boundary  and  high  boundary  values), 

d.  Out-of-type  values  1  to  n, 

e.  Just  under  high  boundary  value,  and 

f.  High  boundary  value. 

•  For  the  invalid  subdomain: 

a.  Just  below  low  boundary  value,  and 

b.  Just  above  high  boundary  value. 

The  actual  values  taken,  of  course,  depend  on  the  type  of  the  data  item  in  question.  Out-of¬ 
type  samples  for  an  integer,  for  example,  are  a  decimal,  single  character,  and  list  of  charac¬ 
ter  values.  As  appropriate,  additional  samples  are  taken;  for  integers  that  include  the  value 
zero,  for  example,  a  troublesome  sample  is  taken,  with  value  zero.  Though  not  experienced 
in  the  course  of  this  examination,  the  user  can  define  his  own  samples  and  modify  the  sam¬ 
pling  rules.  For  example,  the  user  can  add  additional  normal  values  and  abnormal  values. 
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A  technique  called  guided  synthesis  is  used  to  combine  samples.  This  method  provides  a 
repeatable  strategy  for  probing  a  two  dimensional  testing  space  where  only  one  data  item 
is  varied  at  a  time,  the  others  being  held  at  a  normal,  or  reference,  value. 


Tdesign  also  creates  the  Test  Case  Data  Base  (TCDB)  which  contains  one  file  per  test, 
designed  for  easy  use  by  test  execution  tools.  Test  design  metrics  provide  all  the  non-sub- 
jective  values  used  for  function  point  calculation.  (The  test  design  metrics  report  is  not 
available  with  the  demonstration  version.  The  trace  report  that  provides  a  cross-reference 
between  software  actions  and  tests  is  similarly  not  available;  this  information  is,  however, 
included  on  a  case-by-case  basis  in  the  test  case  summary  report  that  details  each  generated 
test  case.) 

T  supports  test  execution  by  providing  a  model  for  calculating  test  coverage.  T  treats 
test  coverage  as  including  requirements,  input  domain,  output  range,  and  structure  cover¬ 
age.  These  factors  are  weighted  individually  and  summed  to  produce  an  overall  Testing 
Comprehensiveness  (TC)  measure.  The  user  can  adjust  the  weights  to,  for  example,  reflect 
different  priorities  at  unit,  integration,  and  system  test  levels.  Requirements  and  input  do¬ 
main  coverage  are  automatically  reported  by  T.  Currently,  the  output  range  and  structure 
coverage  must  be  recorded  manually,  and  TC  calculated  manually.  An  on-line  pass/fail  re¬ 
cording  facility  is  under  development  that  will  allow  a  user  to  specify  test  priorities  and 
record  dates  of  execution  and  test  results  to  allow  automating  this  calculation. 


22.2  Observations 

Ease  of  use.  The  demonstration  version  of  T  is  easy  to  install  and  use;  the  difficult  part 
lies  in  creating  the  TSDL  description.  Although  the  demonstration  version  is  limited  to  a 
character-based  menu  user  interface,  the  full  version  of  T  allows  users  a  choice  between 
this  and  a  graphical  user  interface,  command  line  interface,  or  script  interface.  Context-sen¬ 
sitive,  multi-level  on-line  help  is  available. 

Documentation  and  user  support.  The  documentation  provided  with  the  demonstra¬ 
tion  version  of  T  included  a  full  description  of  TSDL  and  was  adequate  for  its  use.  Pro¬ 
gramming  Environments,  Inc.  were  very  helpful  in  answering  questions. 

Problems  encountered.  The  demonstration  version  operated  as  described  in  the  docu¬ 
mentation. 
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22.3  Recent  Changes  and  Planned  Additions 

A  new  component  is  now  being  delivered  with  T.  Called  Tprepare,  this  component  pro¬ 
vides  for  rule-based  preparation  of  test  documentation.  The  user  defines  his  own  rules  al¬ 
lowing,  for  example,  documentation  to  conform  to  DoD  or  IEEE  standards. 

Several  new  features  are  expected  to  be  released  in  the  first  quarter  of  1993.  Torder,  a 
new  T  component,  will  order  test  cases  and  write  test  scripts  to  handle  the  necessary  test  set 
up  and  clean  up.  Another  component,  Quantifier,  will  take  user-supplied  test  results  to  au¬ 
tomatically  calculate  and  report  on  the  TC  coverage  measure.  T  will  also  support  generation 
of  an  operational  profile  to  support  Musa’s  reliability  assessment. 

An  interface  to  the  AutoTester  test  execution  tool  is  under  development. 


22.4  Sample  Outputs 

Figures  22-1  through  22-7  provide  sample  outputs  from  T. 


i 


i 
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/•  This  Software  Description  File,  SDF,  contains  a  partial  */ 
/•  specification.  There  is  enough  information  in  this  SDF  to  •/ 
/*  illustrate  the  STL  and  the  basic  processing  in  T.  There  is  •/ 
/*  not  enough  information  in  this  SDF  to  specify  a  complete  «/ 
/*  lexical  analyzer  generator  or  to  demonstrate  T  completely.  •/ 


S_paaket 

has  subject 

has  content  version 

has  description 


Adalex_l 

*Adalex_specification"  ; 
•l"/ 

'This  S_Packet  sentence 
'identifies  the  set  of 
'information  in  this  file. 


H 


(include  <tsdl.std>  /*  This  line  includes  standard  definitions  •/ 

/*  The  standard  definitions  will  help  this  self ,  •/ 
/*  but  they  will  also  causa  soma  extra  definitions  */ 
/*  to  appear  in  the  report  called  verify. rpt  •/ 


/* - / 

/*  -  Begin  Action  Definitions  - •/ 

/* - V 


/* - The  words,  identifier  and  integer,  are  keywords  in  IEEE  1175 - •/ 

/*  -  so  they  definition  will  cause  T  to  generate  comments.  However - */ 

/* - there  are  no  reserved  words  in  the  T  scanner /parser  so  the - */ 

/* - key  words  will  be  processed  oorraotly . - «/ 


Action 

is  actiontype 
is  selected  by 
is  concluded  on 
uses  dataitam 


produces  dataitam 
has  description 


Adalax 
internal; 

'invocation*  ; 

'termination*  ; 
oontaxt_and_laxia 
identifier, 
letter, 
digit, 

letter_or_digit , 
integer, 

deoimal_literal , 
oper ator_symbol , 
lef t_parenthesis , 
right_par an thesis 
combining_option 
a_new_lexical_analyzer , 
Standard_Error_File  , 

■The  soanner  produced  depends  upon  * 
'three  items :  1-  A  data  type,  a  * 
'table,  defining  tokens,  2-  A  * 
■get_next_oharaeter  procedure,  and* 
*3-  A  make_token  procedure.  * 


/*  Patterns  •/ 
/*  Patterns  */ 
/*  Patterns  */ 
/*  Patterns  */ 
/*  Patterns  */ 
/*  Patterns  */ 
/•  Patterns  */ 
/*  Patterns  •/ 
/*  Patterns  */ 


Figure  22-1.  T  Sample  SDF 
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/* - */ 

/* - Begin  Data  Definitions - */ 

/» - V 


Dataitem  context_and_lexicon 

has  placeholder  value  "This  ishould  be  the  input  context  am 

has  description  "On  page  5  of  IDA  Paper  P-2100  " 

"the  specification  for  Adalex  " 

’□anas  a  eontext_dause,  a  " 

•generie_for*al_part,  a  separate  " 
"parent_name  and  a  lexicon.  The  * 
"named  items  are  not  defined  in  " 
"P-2100  so  they  are  illustrated  " 

”  in  this  file  with  a  placeholder  " 
"or  to-be-defined  value."  . 


Dataitem  identifier  is  an  instance  of  datatype  identifier_t  . 


Datatype 

is  datatypeclass 
has  values 
has  valid  subdomain 
has  invalid  subdomain 
has  description 


identifier_t 

string; 

//lA-Za-z) 1, 64_[A-Za-zO-9] 1, 64//  ; 

as_specified; 

abnormal; 

"P-2109  did  not  define  the  " 

"allowed  minimum  or  maximum  " 

"of  identifiers.  So  a  minimum  of" 

"one  character  and  a  maximum  of  " 
"128  characters  was  assumed.  " 


Dataitem  letter  is  an  instance  of  datatype  letter_t  . 
Datatype  letterjt 

is  datatypeclass  string; 

has  values  //[A-Za-z]//  ; 

has  valid  subdomain  as_specified; 

has  invalid  subdomain  abnormal; 


Dataitem  digit  is  an  instance  of  datatype  digit_t  . 
DataType  digit_t 

is  datatypeclass  character  ; 

has  values  //I0-9J//  ; 

has  valid  subdomain  as_specifiad; 

has  invalid  subdomain  .  abnormal. 


Dataitem  letter_or_digit  is  an  instanoe  of  datatype  letter_or_digit_t  . 
DataType  letter_or_digit_t 


is  datatypeclass 
has  values 
has  valid  subdomain 
has  invalid  subdomain 


character; 
//[A-Za— I0-9J//; 
as_specified; 
abnormal. 


Figure  22-1  continued:  T  Sample  SDF 
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Dataitem  integer  is  an  instance  of  datatype  integer_t  . 
Datatype  integer_t 

is  datatypeoiass  character; 

has  values  //CO-911, 64//; 

has  valid  subdomain  as_specified; 

has  invalid  subdomain  abnormal. 


Dataitem  decimal_l iter al  is  an  instance  of  datatype  deaimal_literal_t 
Datatype  deoimal_literal_t 

is  datatypeoiass  string; 

has  values  //10-9U, 16.  [0-910,16//; 

has  valid  subdomain  .as_spacifiad; 

has  invalid  subdomain  abnormal. 


Dataitem  operator_symbol 
Datatype 

is  datatypeoiass 
has  values 
has  valid  subdomain 
has  invalid  subdomain 


is  an  instance  of  datatype  operator_symbol_t  . 
operator_symbol_t 
string; 

"+",  •/■; 

as_specified; 

abnormal; 


Dataitem 

has  fired  value 


lef t_par enthes is 
■<’/ 


Dataitem 

has  fixed  value 


right_peren thesis 
')*/ 


Dataitem  oombining_option  is  an  instance  of  datatype  oombinlng_optlon_t . 


Datatype 

is  datatypeoiass 
has  values 
has  valid  subdomain 
has  invalid  subdomain 
has  description 


combining_option_t 

string; 

"is  copied" ,  "is  separata",  "is  generic"  ; 
as_speeified  ; 
abnormal  ; 

"The  combining  option  tells  Axial  ex" 

"how  to  package  the  generated  ■ 
"scanner. ■ 


Dataitem 

has  fixed  value 


e_new_lexioal_eaaly*er 

"Ada  souroe  ooda  for  a  scanner"  ; 


Dataitem 

has  fixed  value 


Standaxd_Xrzor_rile 
"A  report  of  errors  deteeted" 


Dataitem 

has  fixed  value 
has  description 


BOS 

"BOS  is  TSUI  when  the  last  obaracter  is  r  sachet 
"BOS  means  Snd__of _streaa .  EOS  is  " 

"FALSE  as  long  as  there  are  more  " 

"characters  in  the  input  stream.  ”; 


Figure  22*1  continued:  T  Sample  SDF 
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T  Software  Description  Verification  Version  3 . 0  < Restricted) 
Copyright  (C)  1987-1993  Programing  Environments ,  Inc. 


Translation 


1 

•line  1  'sdf' 

13 

/*  This  Software  Description  rile 

,  SDF,  contains  a  partial 

'/ 

13 

/*  specification.  There  is  enough  information  in  this  SDF  to 

*/ 

14 

/•  illustrate  the  STL  and  the  basic  processing  in  T.  There  is 

V 

15 

/•  not  enough  information  in  this 

SDF  to  specify  a  complete 

'/ 

16 

/•  lexical  analyser  generator  or  to  demonstrate  T  completely. 

*/ 

31 

S_packet 

Adalex_l 

33 

has  subject 

■ Adalex_specif ication * 

i 

33 

has  content  version 

*1*; 

34 

has  description 

‘This  S_Packet  sentence 

■ 

35 

'identifies  the  set  of 

■ 

36 

“information  in  this  file. 

n 

37 

38 

39 

lllne  1  '/eval/tcode/tsdl . std* 

1 

•noecho 

143 

•line  30  *sdf  ' 

30 

/•  The  standard  definitions  will  help  this 

sdf  , 

•/ 

174 

Dataitem 

next_character 

175 

has  fixed  value 

'The  output  character 

should  equal  the  1 

176 

177 

<End  of  Flle> 


-  finished  translation  with  46  re cognisable  3 SDL  sentences  out  of  46 

-  saving  description  data  base 

translator  messages 
4  comment  messages 


Interpretation 

sjpaoketi  Adalex_l 
unitdatei  Fri  Oct  9  13 1 13 >49 

1993 

Evaluation 

Aooept  Reject  Name 

»*•  Malax 


Figure  22-2.  T  Software  Description  Verification 
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Extra  dataitams 
EOS 

next_character 

Extra  datatypes 
Bit 

BitSstring 

Digit 

LocalPhone 

PriotableASClI 

USPhone 

ZipCodeS 

ZipCode9 

boolean_integer 

boolean_string 

Extra  states 
<none> 

Extra  statetransitions 
<none> 


Figure  22*2  continued:  T  Software  Description  Verification 


T  Software  Description  Metrics  Version  3.0 
Copyright  (C)  1987-1992  Programing  Environments,  Inc. 

■_packet  s  Adalex_l 

unitdate :  Fri  Oct  9  13: 13 >49  1992 


Total 

Extra 

Verified 

Unverified 

Dynamic 

1 

0 

1 

0 

action(s) 

0 

0 

0 

0 

s tatetrans it ion ( s ) 

Total 

Extra 

Verified 

In  Out 

Unverified 
In  Out 

Static 

0 

0 

0  0 

0 

0 

condition! s) 

15 

2 

11  .  2 

0 

0 

dataitem(s) 

18 

10 

8  0 

0 

0 

datatype! s) 

0 

0 

0  0 

0 

0 

state( s) 

Deficiencies :  0 

inconsistencies :  0 


Figure  22-3.  T  Software  Description  Metrics 
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T  Design  Rule  Verification  Version  3.0 

Copyright  (C)  1987-1992  Programing  Environments,  Inc. 


Translation 

1  /*  T  Design  Rule  Generation  Version  3 . 0 

2  ••  Copyright  (C)  1987-1992  Programming  Environments,  Znc. 

3  */ 

4 


5 

T_Packet 

tpaoket 

6 

7 

e  packet 

Adalex_l 

l 

8 

9 

Local 

context_and_lexicon  . 

10 

11 

Local 

left_parenthesis  . 

12 

13 

Local 

right_parenthesis  . 

14 

15 

CombinationRule 

CR0001 

16 

action 

Adalex; 

17 

singular 

context_and_lexicon , 

18 

identifier, 

19 

letter. 

20 

digit. 

21 

letter_or_digit, 

22 

integer , 

23 

decimal_literal , 

24 

operator_symbol , 

25 

lef t_parentheeis , 

26 

right_parenthesis , 

27 

cambining_opt ion ; 

28 

, 

29 

30 

SelectionRule 

SR0001 

31 

datatype 

identifier_t; 

32 

reference 

TBD; 

33 

valid 

asspecified 

34 

with  function,  boundary,  debug; 

35 

• 

36 

37 

SelectionRule 

SR0002 

38 

datatype 

letter_t; 

39 

reference 

TBD; 

40 

valid 

as_specified 

41 

with  function,  boundary,  debug; 

42 

. 

43 

44 

SelectionRule 

SR0003 

45 

datatype 

digit_t; 

46 

reference 

TBD; 

Figure  22*4.  T  Design  Rule  Verification 
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47 

valid 

&s_specified 

48 

with 

function,  boundary. 

debug; 

49 

, 

• 

50 

51 

SelectionRule 

SR0004 

52 

datatype 

letter_or_di  git_t  ; 

53 

refarance 

TBD; 

54 

valid 

as_specified 

55 

with 

function,  boundary. 

debug; 

• 

56 

, 

57 

58 

SelectionRule 

Sfl0005 

59 

datatype 

integer_t; 

60 

reference 

TBD; 

61 

valid 

as_specified 

• 

62 

with 

function,  boundary. 

debug ; 

63 

, 

64 

65 

SelectionRule 

SR0006 

66 

datatype 

deeiaal_litaral_t ,• 

67 

reference 

TBD; 

68 

valid 

as_specified 

• 

69 

with 

function,  boundary. 

debug; 

70 

. 

71 

72 

SelectionRule 

SR0007 

73 

datatype 

operator_sy»bol_t ; 

74 

reference 

TBD; 

# 

75 

valid 

as_#peoif led 

76 

with 

function,  boundary , 

debug ; 

77 

« 

78 

79 

SelectionRule 

SR0008 

80 

datatype 

ooobin i ng_opt ion_t ; 

0 

81 

reference 

TBD; 

82 

valid 

as_specified 

83 

vith 

function,  boundary, 

debug; 

84 

, 

85 

86 

pe_aark 

-  finished  translation  with  13  recognizable  TDRL  sentences  out  of  13 


Znterpratation 


sjpacket:  Adalex_l 

usitdate  >  Fri  Oct  9  13:13:49  1993 

t_packet :  tpacket 

oaaadata:  Fri  Oct  9  13:17:42  1992 

-  saving  rulas  in  tast  design  data  basa 

Figure  22-4  continued:  T  Design  Rule  Verification 


22-12 


PART  II 


T 


T  Test  Catalog  Version  3.0 

Copyright  (C)  1987-1992  Programing  Environments,  Inc. 

s  pacXat :  Adalex_l 

unitdate :  Fri  Oct  9  13 : 13 i 49  1992 

t jpacket :  tpacket 

casedate;  Fri  Oct  9  13:17:42  1992 

Case  Purpose  (  +  exercises  action,  -  fails  to  exercise  action) 

0001  +  action  Adalex 

state  <unspecified> 
dataitem  all  at  reference 

0002  +  action  Adalex 

state  <unspecified> 
dataitem  all  at  low  boundary 

0003  +  action  Adalex 

state  <unspecified> 
dataitem  all  at  high  boundary 

0004  +  action  Adalex 

state  <unspecified> 

dataitem  identifier  (valid  as_specified  low_bound) 

0005  +  action  Adalex 

state  <unspecified> 

dataitem  identifier  (valid  asspecified  high_bound) 

0006  +  action  Adalex 

state  <unspecified> 

dataitem  identifier  (valid  as_specif ied  comp  [2]  ..reference) 

0007  +  action  Adalex 

state  <unspecified> 

dataitem  identifier  (valid  as_specif ied  coop (2] ..reference) 

0008  +  action  Adalex 

state  <unspecified> 

dataitem  identifier  (valid  as_speoified  comp [2 preference) 


0095  +  action  Adalex 

state  <unspecified> 

dataitem  ccmbining_option  (invalid  out_of_type  out_of_type_3 ) 
-  saving  cases  in  test  design  data  base 


Figure  22-5.  T  Test  Catalog 
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1  Semple  Generation  Version  3.0 

Copyright  (C)  1987-1993  Programing  Environments,  Inc. 
s_packet :  Adalex.l 


unitdmtei 

Fri  Oot 

9  13 i 13 i 49  1993 

t_packet i 

t  packet 

caaedatei 

Fri  Oot 

9  13 i 17 i 43  1993 

coebining_option 

Index 

SubDcaain  Eguiv .  Class 

Label 

Value 

[ 

11 

valid 

aa_speoified 

reference 

*is  separate* 

l 

3] 

valid 

as.specifiad 

lov.bound 

•is  copied* 

l 

31 

valid 

ae_apecified 

high_bound 

■is  generic* 

l 

4] 

invalid 

not_in_liat 

not_in_ll*t 

*  <not_in_list>  * 

[ 

51 

invalid 

out_of_type 

out_of_type_l 

9 

t 

61 

invalid 

out_of_type 

out_o f _type_3 

9.9 

t 

71 

invalid 

out.of.type 

out_of_type_3 

'a' 

context  and lexicon 

Index 

SubDoaain  Equiv.Claae 

Label 

Value 

t 

11 

valid 

placeholder 

reference 

•This  ishould  be  the  input  conti 

declmal_literal 

Index 

SubDcaain  Equiv.Claae 

Label 

value 

( 

11 

valid 

as.specifiad 

reference 

*012.01* 

t 

3] 

valid 

as.spaoified 

lov.bound 

*3.* 

I 

31 

valid 

aa_apeci£ied 

hlgh_bound 

*4567890123456789 . 2345678901234! 

I 

4] 

valid 

ae_apeci f led 

coup [3] _reference 

•0.89* 

( 

51 

valid 

as.specifiad 

°o«P(3]_reference 

*12.01* 

I 

6) 

valid 

as.speclf led 

caap I 3 ] _ref erence 

*345678901234567.23* 

I 

71 

valid 

as.speoified 

caap [2] .reference 

*  8901234567890123 . 45* 

[ 

•1 

valid 

aa_apeoified 

caap [ 3 ] .lov.bound 

*456.* 

l 

*1 

valid 

as.apeaif ied 

aoap { 3 ] .lov.debug 

*789.6* 

C 

101 

valid 

as.speoified 

caap [3 ] _hlgh_debug 

*012.789012345678901* 

I 

HI 

valid 

as.spaoified 

ocap[3]_high_bound 

*345 . 2345678901234567  * 

I 

131 

invalid 

not_tn_liat 

ocap l 0 ] _dropped 

*.89* 

[ 

131 

invalid 

not.in.list 

ocap [1] .dropped 

•67801* 

[ 

141 

invalid 

not_in_li»t 

ocap [ 0 ] _below_bounda 

*.33* 

[ 

151 

invalid 

not_in_liet 

ocap { 0  J  .above.bounds 

*90123456789013345.45* 

t 

161 

Invalid 

not.in.list 

ocap  [lj.belcrw.bounds 

*67867* 

[ 

171 

invalid 

not_in_llat 

ocap (1] _above_bounda 

*901. .89* 

l 

Ml 

invalid 

not_ln_llat 

oaap[3]_above_bounds 

*  234 . 01234567890123456* 

[ 

1»1 

invalid 

not_in_liat 

caap [0] .wrong 

*  1  * . 78* 

t 

30] 

Invalid 

not_in_liet 

coap[l]_vrong 

*567  90* 

f 

31J 

invalid 

not.in.list 

ocap [3] .wrong 

*890.  I*#|%6'()*+,-./* 

[ 

33] 

invalid 

out.of.type 

out.of.type.l 

9 

I 

33] 

invalid 

out_o£_type 

out.of.type.! 

9.9 

[ 

34] 

invalid 

out.of.type 

out_of_type_3 

'a' 

Figure  22-6. T  Sample  Generation 
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T 

digit 

- 

Index 

SubDoaain  Equiv. Class 

Label 

Value 

[ 

1) 

valid 

as_epecified 

ref arenas 

'O' 

t 

2] 

valid 

*«  specified 

lov_bound 

'1' 

[ 

3] 

valid 

as_specifled 

high.bound 

'2' 

t 

4] 

invalid 

not_in_list 

conp [ 0 ] _dr opped 

f  t 

t 

51 

invalid 

not  in  list 

coap [0] .wrong 

t  r 

t 

61 

invalid 

out_of_type 

out  of  _type_l 

9 

( 

71 

invalid 

out.of _  type 

out  of  type.  2 

9.9 

t 

81 

invalid 

out_of_type 

out_of_type_3 

■ <not_in_list>  * 

identifier 

Index 

SubDoaain  Equiv. Class 

Label 

Value 

l 

H 

valid 

as.,  specif  led 

reference 

"ABC.012* 

[ 

2] 

valid 

asspecif led 

lov_bound 

*D_3* 

[ 

31 

valid 

as.spealfied 

high_bound 

*  EFGHI  JKLMNOPQRSTUWOm  abodef  ghi 

t 

41 

valid 

as.spealfied 

coop [ 2 ] _r ef erenoe 

■Q.678* 

[ 

51 

valid 

as_specified 

conp 1 2 ] _ref er once 

■RS.9AB* 

[ 

61 

valid 

as.spealfied 

coap (2]  _ref erenoe 

"TUVMXYZabcdef ghljklanopqrstuvwi 

> 

( 

71 

valid 

asspecif led 

coap  [2] _ref erenoe 

*  ef ghi jklanopqr stuvwxyzABCDEFGHl 

[ 

8] 

valid 

as.specified 

coap  [2]  low  bound 

•grs.I" 

t 

91 

valid 

as_apeoified 

caep[2]_lcrv_debug 

•tuv.JK* 

t 

101 

valid 

as_specified 

coap [2] _high_debug 

‘wxy.LMNOPOKSTUVHXXSabcdefghijU 

C 

HI 

valid 

as_specified 

coap [ 2 ] _high_bound 

*  zAB.NNOPQXSTUVMZTZabodaf ghi jkls 

l 

121 

invalid 

not  in  list 

coap  (0] .dropped 

*_0PQ* 

[ 

131 

invalid 

not_in  list 

coap [11  dropped 

" CDERST  * 

[ 

141 

invalid 

not_in_list 

coap ( 2 ] .dropped 

•wa.* 

> 

[ 

151 

Invalid 

not_in_list 

oaap[0)_belov_bounda 

*_UVW" 

[ 

161 

invalid 

not_in_list 

coap [ 0 ] .above.bounds 

*  UXLMMOPQRSTUVHXXZabcdef  ghi  jkls 

t 

171 

invalid 

not_in  list 

coap [ 1 ) .below.bounda 

"VWXabc" 

( 

181 

invalid 

not.in  list 

ooap[l]  .above.bounds 

"YZa  def 

[ 

191 

invalid 

not_ln_list 

ooap[2]  belov  bounds 

"bod." 

[ 

20] 

invalid 

not_in_list 

ooap  [ 2 ]  above  bounds 

*efg_gbijklanopqrstuvwxyx012345C 

[ 

21] 

invalid 

not_in_list 
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Figure  22-6  continued:  T  Sample  Generation 
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Figure  22-6  continued:  T  Sample  Generation 
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T  Test  Casa  Definitions  Vara ion  3.0 

Copyright  (C)  1987-1993  Programing  Environments,  Ine. 


a jpaokat » 
unltdata : 
t  pact at : 
oasedate t 


Adalex_l 

rri  Oct  9  13:13:49  1992 
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Fri  Oct  9  13:17:42  1992 


INDEX 

CASBtAME 

ecekcises 

IN  STATE 
REASON 
INPUT  DATA 
Name  -  Value 


0001 

01000001 

Ad&lex 

<unapeoif iad> 
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oontaxt_and_lexicon 

-  'This  ishould  be  the  input  context  and  lexicon  description . " 

identifier 

-  * ABC_012  * 

latter 

-  "A* 

digit 

-  '0' 
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-  'O' 

integer 
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- 
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-  •(• 

right_paren thesis 
-  ■)• 
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—  "is  separata* 
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BID  ST  termination 

OUTPUT  DATA 
Mama -  Value 
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-  'Ada  source  code  for  a  scanner* 

Standard_srror_Pile 

-  *A  report  of  errors  detected* 


TSANSITION  <none> 


Figure  22*7.  T  Test  Case  Definitions 
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-  'This  ishould  ba  tha  input  context  and  lexicon  description . ' 
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latter 

■ -  *B* 

digit 

-  >!* 

letter  or_diglt 

-  <1‘ 

integer 

-  '3' 

decimal  literal 
*3.* 

operator_*yabol 
-  ■+■ 

left_paren thesis 

- 

right_parenthe*i* 

- 

ooablnlng_optlon 

-  "ia  copied* 


a_new_lexioal_*n*ly*ar 

-  "Ada  source  oode  for  a  scanner* 

Standard JError_FUe 

—  *A  report  of  errors  detected* 
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Figure  22-7  continued:  T  Test  Case  Definitions 
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PART  II  T-PLAN 

23.  T-PLAN 

T-PLAN  is  a  test  planning  and  modeling  method  with  an  associated  PC-based  tool.  The 
T-PLAN  method  is  documented  in  a  series  of  manuals  that  deal  with  strategic  test  planning, 
resource  organization,  and  management.  These  manuals  include  guidelines  that  help  a  user 
to  structure  and  partition  testing  into  manageable  pieces  dependent  on  a  series  of  factors 
such  as  risk  and  budget.  The  associated  tool  supports  planning,  organizing,  and  document¬ 
ing  test  activities.  The  following  evaluation  focuses  solely  on  the  T-PLAN  tool. 

Aimed  primarily  at  functional  testing  at  the  system  level,  the  T-PLAN  tool  can  be  used 
for  testing  activities  throughout  development  In  addition  to  supporting  the  planning  and 
documentation  of  test  activities,  it  provides  statistical  analyses  to  monitor  these  activities. 
Change  impact  analysis  identifies  those  parts  of  a  system  under  test  that  are  affected  by  a 
modification. 

23.1  Tool  Overview 

T-PLAN  is  marketed  by  Software  Quality  Assurance,  Ltd.  in  England.  This  company 
will  examine  a  customer’s  testing  requirements  to  develop  an  implementation  plan  for  T- 
PLAN  installation.  This  service  can  include  T-PLAN  customization  through  the  develop¬ 
ment  of  appropriate  test  models  and  data  entry  templates.  Software  Quality  Assurance  also 
provides  strategic  test  planning  consultancy  and  independent  system  testing,  as  well  as 
training  and  seminars. 

The  tool  has  been  available  since  1989  and  has  over  120  users.  It  runs  on  an  IBM  PC, 
or  compatible,  under  either  DOS  (release  3.1  onwards)  or  Unix.  T-PLAN  employs  a  fourth 
generation  language  and  associated  relational  database.  It  can  be  networked  on  all  major 
PC  networks  to  enable  a  team  of  people  to  design,  document,  and  review  testing  details.  At 
the  time  of  evaluation,  T-PLAN  prices  started  at  £9,500.  The  evaluation  was  performed  on 
a  demonstration  copy  of  T-PLAN  version  2.0.  This  demonstration  software  is  fully  func¬ 
tional,  except  (1)  only  a  limited  number  of  records  can  be  added  to  the  supplied  test  data 
base,  and  (2)  test  data  input  and  output  templates  cannot  be  customized. 

T-PLAN  allows  the  user  to  define  the  underlying  test  model,  although  the  model  de¬ 
fined  by  Software  Quality  Assurance  can  be  used  for  this  purpose.  The  structure  of  the  re¬ 
sulting  test  model  is  recorded  in  the  T-PLAN  Test  Dictionary.  Consequently,  with  T- 
PLAN,  die  test  process  starts  with  establishing  the  structure  of  the  underlying  test  dictio- 
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nary.  The  desired  structure  is  typically  determined  through  a  modeling  activity  that,  based 
on  documentation  such  as  the  functional  requirements  description,  specifies  the  expected 
behavior  of  the  system  under  pre-determined  conditions.  It  is  specified  in  terms  of  func¬ 
tions,  inputs,  and  outputs,  called  test  entities.  Specifically,  the  following  object  types  are 
recognized: 

•  Test  Specification.  The  highest  level  entity  that  defines  the  overall  test  plan. 

•  System  Function.  A  condition  to  be  tested,  given  at  the  level  where  individual  test 
conditions  can  be  identified  for  each  function.  Where  necessary,  system  functions  can 
be  equated  to  system  properties  such  as  performance,  recovery,  or  stress. 

•  Source  System  Input  Content  and  format  of  the  input  data  required  for  testing. 

•  System  Output/Data  Profile.  The  contents  and  format  of  expected  output  data. 

•  System  Program.  An  optional  entity  that  allows  storing  program  reference  ^  and  cross- 
referencing  these  to  functions  and  files  so  that  the  affect  of  a  program  change  on  test¬ 
ing  can  be  assessed. 

•  System  File.  An  optional  entity  that  allows  storing  file  references  and  cross-referenc¬ 
ing  these  to  functions  and  programs  so  that  the  affect  of  a  program  change  on  files  can 
be  assessed. 

•  Service  Queries.  A  record  of  the  complete  history  of  a  change. 

Once  the  dictionary  structure  has  been  established.  Test  Conditions  are  entered  for  the 
identified  functions.  These  provide  descriptions  of  the  functional  conditions  that  are  re¬ 
quired  to  be  tested.  Usually,  they  drive  the  design  of  test  inputs  and  expected  outputs.  Test 
Conditions  are  captured  with  a  unique  reference  and  structured  into  Test  Sets  via  a  Function 
Reference;  further  grouping  of  Valid  and  Invalid  categories  can  be  assigned.  Conditions  re¬ 
lating  to  particular  releases  or  versions  of  a  system  can  also  be  grouped  together.  Test  Con¬ 
ditions  are  cross-referenced  to  test  inputs  and  expected  results.  Additionally,  common  Test 
Conditions  that  are  to  be  centrally  held  and  reused  can  be  defined. 

Test  input  data  is  created  via  user-tailorable  templates.  These  are  designed  to  match 
source  system  inputs  (usually  screens),  thereby  giving  the  feel  and  look  of  using  the  actual 
system.  Special  templates  for  “No  Screen  Data”  testing  can  be  used  for  scripting  and  allow 
script  narrative  to  be  cross-referenced  to  Test  Conditions.  Scripting  can  be  combined  with 
input  templates  as  required. 

The  user  also  defines  templates  for  expected  output  data,  called  Output  Data  Profiles. 
These  data  profiles  are  designed  to  match  system  outputs  or  to  give  a  logical  pointer  to 
where  output  data  is  expected.  Each  represents  a  view  of  a  record,  file,  or  report  of  a  spe¬ 
cific  data  entity  in  a  given  time-frame.  Thus,  the  history  of  a  data  entity  can  be  recorded  and 
tracked  through  the  entire  test  plan.  A  special  “No  Profile  Data”  template  is  available  for 
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capturing  expected  results.  This  template  allows  the  user  to  give  a  narrative  expected  result 
“Check  List”  and  cross-reference  it  to  Test  Conditions.  As  before,  this  type  of  scripting  can 
be  combined  with  the  output  data  profile  templates  as  required. 

The  dictionary  employs  cross-referencing  at  both  the  test  entity  level  and  the  data  level. 
Various  static  analysis  functions  make  cross-reference  listings  available  to  the  user.  This 
cross-referencing  enables  T-PLAN  to  analyze  the  impacts  of  changes  on  Systems  Func¬ 
tions,  Inputs  and/or  Outputs.  By  identifying  areas  of  the  system  that  are  directly  affected  by 
a  change,  and  areas  functionally  dependent  on  the  area  being  changed,  T-PLAN  helps  to 
identify  regression  testing  requirements. 

The  test  dictionary  itself  consists  of  test  specifications  that  contain  the  information  re¬ 
quired  to  test  and  check  test  results  for  a  given  part  of  the  system.  A  test  specification  can 
contain  one  or  more  test  paths,  where  a  test  path  is  a  collection  of  tests  that  form  a  test  run. 
Test  paths  can  be  thought  of  as  a  timeframe  in  which  a  particular  set  of  tests  must  be  run. 
They  can,  in  turn,  be  grouped  into  test  cycles  to  define  a  series  of  dependent  test  timeframes 
and/or  test  specifications.  A  log  of  test  data  and  dated  records  of  test  events  are  subsequent¬ 
ly  stored  against  a  test  specification  to  allow  a  history  of  testing  activities  to  be  maintained. 
By  mapping  test  specifications  to  software  components,  the  dictionary  provides  for  trace- 
ability  of  test  data  to  the  software  under  test  If  required,  test  data  may  be  linked  across  test 
specifications  and  its  history  tracked  via  Data  Profiles.  Traceability  is  also  provided  at  the 
entity  level,  that  is,  between  test  specifications  and  functions,  inputs,  and  outputs.  Test 
schedule  activities  are  defined  in  terms  of  a  system,  phase,  package,  and,  optionally,  soft¬ 
ware  build  parameters.  They  are  separated  under  three  categories:  test  preparation,  testing, 
and  regression  testing,  each  of  which  has  an  associated  review  process.  In  each  case,  the 
user  can  specify  who  is  responsible  for  the  activity,  an  estimate  of  the  number  of  days  re¬ 
quired,  the  actual  days  completed  so  far,  and  an  estimate  of  days  outstanding.  This  data  is 
used  to  produce  test  progress  reports.  These  reports  include  figures  of,  for  example,  per¬ 
centage  completed  against  schedule.  The  management  reports  provide  information  at  the 
system,  build,  or  package  and  phase  levels.  A  Test  Management  Summary  reports  on  the 
status  of  testing  with  respect  to  test  paths. 

Changes,  and  change  control  information,  can  also  be  recorded  against  test  specifica¬ 
tions.  The  change  control  management  system  utilizes  “Service  Queries.”  The  user  can 
record  queries  and  errors,  whether  or  not  they  sponsor  a  change,  as  service  queries.  This 
enables  tracking  the  complete  history  of  a  change,  including  description,  prioritization,  es¬ 
timates,  actual  and  outstanding  effort,  query  or  error  classification,  and  software  library  re- 
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lease  information.  Management  reports  provide  for  monitoring  the  progress  of  changes.  For 
example,  reports  analyze  the  totals  of  errors  or  queries  by  classification,  the  frequencies  or 
errors  and  frequency  of  clear-up,  as  well  as  the  percentage  completed  and  outstanding  effort 
to  complete  changes.  This  type  of  information  can  be  used  to  check  test  progress  and  assist 
in  planning  future  projects. 


23.2  Observations 

Ease  of  use.  The  tool  provides  a  menu  driven  interface,  where  the  user  uses  the  key¬ 
board  to  make  selections  and  input  data.  Database-type  search  and  display  operations  are 
provided  for  viewing  the  test  dictionary  contents.  There  is  no  graphics  capability.  The  on¬ 
line  help  facility  provides  a  description,  derivation  formula,  and  cross  reference  to  user 
manuals. 

Templates  are  used  throughout  to  facilitate  data  input  Since  the  user  may  customize  ex¬ 
isting  templates,  and  create  new  templates,  this  provides  some  tailorability  to  the  tool.  To 
keep  stored  test  data  in  line  with  the  templates,  when  a  template  is  changed,  all  test  data 
stored  for  the  template  is  automatically  reformatted  to  match  the  new  template.  In  certain 
circumstances,  data  captured  via  input  templates  can  be  exported  in  external  file  formats  to 
aid  in  setting  up  file  data  needed  for  testing.  The  file  formats  available  include  DIF,  dBase, 
Lotus  1-2-3.  Conversely,  data  in  these  formats  can  be  imported  into  T-PLAN. 

Another  helpful  feature  is  the  ability  to  create  central  data  profiles.  These  are  centrally 
held  tables  of  data  that  can  be  accessed  from  input  and  output  templates.  This  not  only  re¬ 
duces  the  need  to  rekey  repeated  data  but  reduces  the  possibility  of  discrepancies  between 
repeated  pieces  of  data.  The  tool  comes  with  one  centrally  held  file  already  set  up  to  hold 
a  copy  of  Error  Messages  and  Codes.  (System  error  messages  file  data  can  be  imported  di¬ 
rectly  into  the  tool.) 

Documentation.  Only  the  documentation  for  the  demonstration  version  of  the  tool  was 
supplied.  This  was  adequate  for  its  purpose.  The  full  version  of  T-PLAN  is  accompanied 
by  four  volumes  relating  to  test  management,  strategy,  test  modeling,  and  technical  issues. 
Installation  was  straightforward. 

Problems  encountered.  The  tool  operated  as  described  in  the  documentation. 
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23.3  Recent  Changes  and  Planned  Additions 

An  interface  from  T-PLAN  to  Direct  Technology  Ltd.  Automator-QA  is  now  available. 
This  allows  test  input  data  stored  in  T-PLAN  to  be  “played”  directly  into  the  software  under 
test  via  Automator  test  scripts.  An  interface  to  automatically  feed  test  log  information  back 
into  T-PLAN  is  under  development. 


23.4  Sample  Figures 

Figures  23-1  through  23-17  provide  sample  outputs  from  T-PLAN. 
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FUNCTIONAL  CONDITION  LIST  AS  AT  04/01/80  VERSION  1.1 


FUNCTION  REF 
FUNCTION  NAME 
SYSTEM 

OVERALL  FUNCTION 
RELEASE  NO. 


t  FXI  FOR  TEST  SPEC  REF: 

:  Foreign  Exchange  Input 
:  IBS 

:  FX  On-line  Deal  Processing 

1.0  INCLUDE/EXCLUDE  I 

INVALID/VALID  :  I 


CONDITION  PATH 

NO. 

Deal  Type  not  completed  01 

Deal  Type  not  valid  code  01 

Deal  Date  later  than  value  date  01 

Forward  valued  Deal  Date  before  value  date  01 


'  DEV  DOC 
REFS 

SCSD2 . 1 
SCSD2.1 
SCSD2 . 3 
SCSD2 . 3 


FUNCTIONAL  CONDITION  LIST  AS  AT  04/01/80  VERSION  1.1 


FUNCTION  REF 
FUNCTION  NAME 
SYSTEM 

OVERALL  FUNCTION 
RELEASE  NO. 


FX2  FOR  TEST  SPEC  REF:  FA2 

Foreign  Exchange  2nd  Authorise 
IBS 

FX  On-line  Deal  Processing 


1.0  INCLUDE/EXCLUDE  I 

INVALID/VALID  :  I 


CONDITION 


DEV  DOC 
REFS 


Attempt  to  2nd  Authorise  a  Deal  that  05  SCSD2.3 

has  not  been  1st  Authorised 

Attempt  to  2nd  Authorise  a  Deal  without  05  SCSD2.3 
appropriate  security  level 

Attempt  to  2nd  Authorise  a  Deal  that  SCSD2.3 

has  already  been  2nd  Authorized 

Attempt  to  2nd  Authorise  a  Deal  that  05  SCSD2.3 


16  J  Interest  Arbitrage  Deal  (Deal  Type  FA) 


SCSD2.3 


PERCENTAGE  OF  TESTS  TO  BE  INCLUDED  84.004 


Figure  23-1.  T-PLAN  Test  Model  Functional  Condition  List  Report 
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- - - 

|  INPUT  REFERENCE  EIN  | 

+ - - 


|  FUNCTION  KEY  :  ENTER 
|  TEST  SPEC  REF  FIN  PATH  01 


- + 

I 

- + 

SEQUENCE  NO.  0010  j 

- f 


DLIN  X  FX  DEAL  INPUT 

DEAL  TYPE  VN  DEAL  DATE  271290 


COUNTERPARTY 
PURCHASED  CURRENCY 
SOLD  CURRENCY 
CROSS  RATE 
STERLING  RATE 
VALUE  DATE 
CONFIRMATION  METHOD 
DEALING  METHOD 
SWAP  BASE  RATE 
A-TYPE  DEAL  SPOT  RATE 
OUR  RECEIVING  AGENT 
OUR  PAY IMG  AGENT 
THEIR  RECEIVING  AGENT 
BENEFICIARY  ACCOUNT 
FREE  FORMAT 


00109 

DEM 

NOK  AMOUNT  225,000 

4.444444 

2.934 

271290  OPTION  FROM  TO 

S  PAYMENT  METHOD  N 

10  COVER  PAYMENT  REQUIRED  N 
CCY  IF  NOT  US$ 
FLAT  CURRENCY 

C  00109HN0101 
A  00109DB0107 


DDMMYY  TERM  0A1P 


DL  X 
A/C  REF. 


50 

NAM 

STR 

57 

NAM 

STR 

59 

NAM 

STR 

70 


SUPPLEMENTARY  PAYMENT  DETAILS  AND  CHARGES 
CODE  NARRATIVE  CCY  D/C  AMOUNT 

SUPPLEMENTARY  PAYMENT  DETAILS 
:  ORDERING  CUSTOMER 


:  "ACCOUNT  WITH"  BANK 


: BENEFICIARY  CUSTOMER 


: DETAILS  OP  PAYMENT 


NAM 

PLC 

ACT/ 

NAM 

PLC 

ACT/ 

NAM 

PLC 


sp  .5 

71  i DETAILS  OF  CHARGES  DIRECT  PAYMENT  METHOD 


TERM  0A1P 
ERRORS/EXPANSIONS 


I 

|  Not* 
4 - 


ERROR  MESSAGE 


TEST  PLAN  NOTES/SCRIPT 

Supplementary  Payments  screen  should  not  appear 


+ 


+ 


Figure  23-2.  T-PLAN  Test  Model  Sample  Print  for  Input  Ref 


23-7 


T-PLAN 
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T-PLAN  TEST  DICTIONARY  AS  AT  04/01/80  VERSION  1.1 
TEST  SPECIFICATION  /  INPUT  REFERENCE  MATRIX 


TEST  SPEC  REFERENCE  :  FIN 


INPUT  REFERENCE 
INPUT  REFERENCE 
INPUT  REFERENCE 


DMN  Deal  Menu 

EEN  Foreign  Exchange  Data  Enquiry 
EIN  Foreign  Exchange  Deal  Input 


T-PLAN  TEST  DICTIONARY  AS  AT  04/01/80  VERSION  1.1 
TEST  SPECIFICATION  /  OUTPUT  REFERENCE  MATRIX 

TEST  SPEC  REFERENCE  :  FIN  Foreign  Exchange  Deal  Input/Sumary  Reports 
OUTPUT  REFERENCE  :  DEL  Deal 

OUTPUT  REFERENCE  :  FBS  FX  Batch  Summary  Report 

OUTPUT  REFERENCE  -.  FIS  FX  Deal  Input  Summary  Report 


INDEX  OF  OUTPUT  REFERENCES  AS  AT  04/01/80  VERSION  1.1 


INPUT  INPUT 

REF  NAME 


DMN  Deal  Menu 

EAl  Foreign  Exchange  Deal  1st  Authorise 

EA2  Foreign  Exchange  Deal  2nd  Authorise 

EAM  Foreign  Exchange  Deal  Amend 

EDE  Foreign  Exchange  Deal  Delete 

ENE  Foreign  Exchange  Deal  Enquiry 


MIN 

STA 


Money  Movement  Input 

General  Deal  Status  Enquiry  Screen 


INDEX  OF  OUTPUT  REFERENCES  AS  AT  04/01/80  VERSION  1.1 


OUTPUT 

REF 

OUTPUT 

NAME 

APL 

Monthly  PtL  Report 

ATP 

Trial  Balance  Report 

BEF 

Bulked  Entry  File 

CAB 

CAN  Average  Daily  Balance  Summary 

CAC 

Customer  Acount 

CCY 

Foreign  Exchange  Rates 

SPA 

System  Parameter  Data 

SWI 

SHIFT  Messages 

Figure  23*3.  T-PLAN  Test  Model  Input  &  Output  References  for  Test  Spec  FIN 
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T-PLAN 


SOURCE  INPUT  REF 
TEST  SPEC  REF 


NO  SCREEN  DATA  TESTING  (NSD) 

EIN  Foreign  Exchange  Deal  Input 

FIN  PATH  NUMBER  :  01  SEQUENCE  NO.  0060 


CONDITION  REF  TEST  NOTES  TESTED  OK? 

FXI2IA  Orderer  Client  -  32345 
FXI21B  Orderer  Client  -  Non  Swift  »< 

FXI21G  Orderer  Client  68213  -  Account  With  Bank  99-00-88 

FXI21H  Account  With  Bank  MADGGG02 

FXI21I  Account  With  Bank  02356 

FXI21J  Account  With  Bank  80197 

FXI21K  Account  With  Bank  Non  Swift  s@ 

FXI21P  Details  of  Payiiant  spaces  -  Details  of  charges  BEF 


Figure  23-4.  T-PLAN  Test  Model  No  Screen  Data  Testing  for  FIN 


► 


OUTPUT  DATA  PROFILE  NAME 

OUTPUT  DATA  PROFILE  REF 
TEST  SPEC  REF 


•  DEAL  DATA  PROFILE 
MAJOR  SUB  TIME 

DP  KEY  KEY  FRAME  (PATH) 
:  DEL  003  00  01 

*  FIN 


DATA  PROFILE 
REF 

DEL0030001 


I 


I 


> 


I 


NOTES 

Deal  should  be  delated  after  batch  overnight  run 

DEAL  NUMBER  Syst  gen  DEAL  TYPE  SN  DEAL  DATE  271290 

COUNTERPARTY  00203 

PURCHASED  CURRENCY  GBP  AMOUNT  10,000,000.00 
SOLD  CURRENCY  USD 

1.735 
1.75 

311290  OPTION  FROM  TO 

S  PAYMENT  METHOD  S 

10  COVER  PAYMENT  IF  REQUIRED  N 
CCY  IF  NOT  US$ 

FLAT  CURRENCY 

C  12973HN0505 
22734HN0202 
45798 


CROSS  RATE 
STERLING  RATE 
VALUE  DATE 
CONFIRMATION  METHOD 
DEALING  METHOD 
SWAP  BASE  RATE 
A-TYPE  DEAL  SPOT  RATE 
OUR  RECEIVING  AGENT 
OUR  PAYING  AGENT 
THEIR  RECEIVING  AGENT 
BENEFICIARY  ACCOUNT 
INHIBIT  ADVICE  TO  RECEIVE? 
71  : DETAILS  OF  CHARGES 


DIRECT  PAYMENT  METHOD 


DEAL  STATUS  01  CONF  STATUS  01  PAYMENT  STATUS  01 

Figure  23-5.  T-PLAN  Test  Model  Output  Print  for  FIN 
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SPECIFICATION  INSTRUCTIONS  AS  AT  04/01/80  VERSION  1.1 
TEST  SPEC  REF  :  FIN  Foreign  Exchange  Deal  Input/Summary  Reports 
PAGE  NUMBER  :  01  AUTHOR  :  THEO  CCUPIER 

TEST  SPECIFICATION  PREREQUISITES  AND  INSTRUCTIONS 


This  Test  Specification  teste  the  following  functionality  in  the  IBM 
system : - 

Foreign  Exchange  Input 
Foreign  Exchange  Enquiry  (Part) 

FX  Batch  Summary  (Part) 

FX  Deal  Input  Summary  (Part) 

Note  -  Only  part  of  some  of  the  above  functions  are  tested  in  this 
Specification 

The  Test  Spec  has  Input  for  the  following  days  in  the  cycle 
01,  05 

The  Test  Spec  has  output  to  be  checked  for  the  following  days 
01,  05 

Before  commencing  testing  for  PATH  1,  ensure  that  the  following  test 
environment  is  in  place 

-  System  Test  Libraries  correctly  loaded 

-  System  Test  Base  Data  files  correctly  loaded 

-  Customer  and  Account  files 

-  Rates  files 

-  User  t  Password  files 

-  System  parameter  file  data  for  Day  1 


TEST  PATH  SUMMARY  REPORT  AS  AT  04/01/80  VERSION  1.1 
TEST  SPEC  REF  :FXN  Foreign  Exchange  Deal  Input/summary  Reports 

PATH  NUMBER  05 

Logon  to  the  system  with  User-id  c  Password  X1AA 
Enter  Script  Input  from  Path  05 

Ensure  that  a  note  is  made  of  all  Deal  Numbers  allocated 
After  Entering  all  data  and  PRIOR  to  running  Day  5  Batch  Reports 
Print  DEAL  file  and  check  against  data  profiles 

After  running  Batch  Reports  for  Path  05  Check  that  Deals  are  correctly 
deleted  from  DEAL  file  as  noted  on  data  profiles 


TEST  SPECIFICATION  TESTING  LOG  BY  PATH 
TEST  SPEC  REF  ;  FIN  Foreign  Exchange  Deal  Input/summary  Reports 
PATH  NUMBER  TEST  DATE 

01  FXN01  19/09/91 

TESTER  :  GAH  RETEST  REQUIRED  :  yes 

COMMENTS 

Service  Query  1  raised 

Figure  23*6.  T-PLAN  Test  Model  Test  Specification  Information  for  FIN 
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T-PLAN 


Cycle  (02)/Path  Overview  Report 

TEST  CYCLE/PATH  OVERVIEW  AS  AT  04/01/80  VERSION  1.1 

CYCLE  /PATH  NUMBER  02 

RUN  SEQUENCE  NO.  :  001 

TEST  SPEC  REF  :  MM1  Money  Movement  let  Authorisation 

cotaams 

Initial  MM  1st  Auth  testing 
set-up  of  deals  t  1st  auth 

TEST  CYCLE/PATH  OVERVIEW  AS  AT  04/01/80  VERSION  1.1 

CYCLE  /PATH  NUMBER  02 

RUN  SEQUENCE  NO.  :  003 

TEST  SPEC  REF  s  FA2  Foreign  Exchange  Deal  2nd  Authorisation 

COMMENTS 

'Base  Data*  setup  for  FX2  Testing 


Cycle  (02)! Path  Summary  Report 


000  BAS  Base  data  set-up 

001  FF1  Forex  Fixed  Deal  1st  Authorisation 
001  MM1  Money  Movement  1st  Authorisation 
001  FA1  Foreign  Exchange  Deal  1st  Authorisation 
001  Dll  Discounted  Items  1st  Authorisation 
002  DEE  general  Deal  Enquiries 

003  FAD  Foreign  Exchange  Deal  Amend/Delete/Write-Off 

003  FAS  Foreign  Exchange  Deal  2nd  Authorisation 


Test  Specification  Input/ Condition  Cross  Reference  Report  for  FIN 

FX112A 
FXI12B 
FZZ12C  * 

FZI13A 

FXI13B 

FXX14A 

FXI14B 

FZZ20A 

TEST  SPECIFICATION  INPUT  BY  PATH  SUMMARY  AS  AT  04/01/080  VERSION  1.1 
TEST  SPEC  REF  :  FIN  Foreign  Exchange  Deal  Input/Summary  Reports 


Figure  23-6  continued:  T-PLAN  Test  Model  Test  Specification  Information  for  FIN 
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PATH  NUMBER  01 

FUNCTIONAL 

INPUT 

CONDITION 

REF 

REF 

SEQUENCE  NO. 

0005 

NOXREF 

DMN 

SEQUENCE  NO. 

0010 

FXI01A 

EIN 

SEQUENCE  NO. 

0020 

FXI01B 

FXI02B 

FXI02C 

FXI02D 

NSD 

SEQUENCE  NO.  0110  EXE 11  SEN 

FXE17 


PATH  NUMBER  01 

|  TESTED  BY  •  DATE  »  CHECKED  BY  *  DATE 


+ - - 

PATH  NUMBER  05  FUNCTIONAL  INPUT 

CONDITION  REP  REF 

SEQUENCE  NO.  0005  DMN 

NOXREF 


PATH  NUMBER  05 


- 

1 

tested  by 

a 

DATE 

a 

a 

DATE 

— 4 

1 

CHECKED  BY 

1 

a 

« 

a 

| 

1 

I 

4 - 

• 

• 

a 

1 

list  Specification  Exception  Report  , 

TEST  SPEC  REF  i  FIN  foitljn  Exchange  Deal  Input/Sunary  Reports 

RELEASE  NO.  1.0  INVALID/VALID  V 

FUNCTIONAL  CONDITION  REFERENCE 

FXI24I 

FXI24J 

1X1150 

FXX15P 

FXI15R 

FXI24H 

FUNCTIONAL  CONDITIONS  NOT  CROSS  REFERENCED  AS  AT  04/01/80  VERSION  1.1 


Figure  23-6  continued:  T-PLAN  Test  Model  Test  Specification  Information  for 
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INDEX  OF  FUNCTION  REFERENCES  AS  AT  04/01/80 


FUNCTION 

REF 


FUNCTION  NAME 


VERSION  1.1 


SYSTEM  :  IBIS 

OVERALL  FUNCTION  :  Call  t  Notice  -  Money  Movement 
MM1  Money  Movement  1st  Authorise 

MM3  Money  Movement  3nd  Authorise 

NNA  Money  Movements  Amend 

MMD  Money  Movements  Delete 

MME  Money  Movements  Enquiry 

MU  Money  Movements  Input 


FUNCTION 

REF 


FUNCTION  NAME 


SYSTEM  :  IBIS 

OVERALL  FUNCTION  :  Position  Enquiries 
PCU  Customer  Positions 

PCY  Currency  Positions 

PDL  Dealer  Positions 

Money  Movement  Enquiry 

INDEX  OF  INPUT  REFERENCES  AS  AT  04/01/80  VERSION  1.1 


INPUT  INPUT 

REF  NAME 


DMA  Deal  Menu 

EA1  Foreign  Exchange  Deal  1st  Authorise 

EA3  Foreign  Exchange  Deal  3nd  Authorise 

EAM  Foreign  Exchange  Deal  Amend 

EDE  Foreign  Exchange  Deal  Delete 


Money  Movement  Enquiry 

INDEX  OF  OUTPUT  REFERZMCES  AS  AT  04/01/80  VERSION  1.1 


OUTPUT  OUTPUT 

REF  NAME 


CZR  CAN  Interest  Rete  Change  Notification 

CSS  CeN  Daily  Summary  Reports 

CUS  Customer 

DDD  DBR  Daily  Detail  Report 

DEL  Deal 


Figure  23-7.  T-PLAN  Test  Dictionary  Function,  Input,  Output  Reference  Index 
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T-PLAN  TEST  DICTIONARY  AS  AT  04/01/80  VERSION  1.1 
TEST  SPECIFICATION  /  FUNCTION  REFERENCE  MATRIX 

TEST  SPEC  REFERENCE  :  CIR  Call  t  Notice  Interest  Rate  Change  Notification 
FUNCTION  REFERENCE  :  CIR  CtN  Interest  Rate  Change  Notification 

T-PLAN  TEST  DICTIONARY  AS  AT  04/01/80  VERSION  1.1 
TEST  SPECIFICATION  /  FUNCTION  REFERENCE  MATRIX 

TEST  SPEC  REFERENCE  :  FIN  Foreign  Exchange  Deal  Input/Suaeary  Reporta 

FUNCTION  REFERENCE  :  PBS  FX  Batch  Sunary 

FUNCTION  REFERENCE  :  FIS  FX  Deal  Input  Sunary 

FUNCTION  REFERENCE  :  FXE  Foreign  Exchange  Enquiry 

FUNCTION  REFERENCE  :  FXI  Foreign  Exchange  Input 

T-PLAN  TEST  DICTIONARY  AS  AT  04/01/80  VERSION  1.1 
TEST  SPECIFICATION  /  INPUT  REFERENCE  MATRIX 

TEST  SPEC  REFERENCE  :  FIN  Foreign  Exchange  Deal  Input/Sunary  Reports 

INPUT  REFERENCE  >  DMN  Deal  Menu 

INPUT  REFERENCE  :  EA1  Foreign  Exchange  Deal  1st  Authorise 

INPUT  REFERENCE  ;  EA2  Foreign  Exchange  Deal  2nd  Authorise 

INPUT  REFERENCE  :  XAM  Foreign  Exchange  Deal  Anend 

INPUT  REFERENCE  ■  EEN  Foreign  Exchange  Deal  Enquiry 

INPUT  REFERENCE  :  EIN  Foreign  Exchange  Deal  Input 

T-PLAN  TEST  DICTIONARY  AS  AT  04/01/80  VERSION  1.1 
TEST  SPECIFICATION  /  OUTPUT  REFERENCE  MATRIX 

test  SPEC  REFERENCE  :  fin  Foreign  Exchange  Deal  input/Sunaary  Reports 

OUTPUT  REFERENCE  :  DEL  Deal 

OUTPUT  REFERENCE  :  FBS  FX  Batch  Susaary  Report 

OUTPUT  REFERENCE  .  FIS  FX  Deal  Input  Suaaary  Report 


Figure  23-8.  T-PLAN  Test  Dictionary  Functions,  Inputs,  Outputs  Used  in  FIN 


CONDITIONS  IMPACTING  ON  OUTPUT  DATA  PROFILES  AS  AT  04/01/80  VERSION  1.1 
ACROSS  ALL  TEST  SPECIFICATIONS 

DATA  PROFILE  KEY  :  DEL002 

PATH  :  01 

TEST  SPEC  :  Foreign  Exchange  Deal  Input/Sunaary  Reports 

DATA  PROFILE  REF  -.  DEL0020001  CONDITION  REF  i  FXI11 
SOURCE  INPUT  XET(SP 
FIN010040EIN 

Figure  23-9.  T-PLAN  Test  Dictionary  Condition  impact  on  Data  Profiles 
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T-PLAN  TEST  DICTIONARY  AS  AT  04/01/80  VERSION  1.1 
FUNCTION  CHANGE  INPACT  ANALYSIS  REPORT 

FUNCTION  REF  :  MME  Money  Movement  Enquiry 


TEST 

SPEC 

REF 

:  MM1 

Money 

TEST 

SPEC 

REF 

:  MM2 

Money 

TEST 

SPEC 

REF 

:  MMA 

Money 

TEST 

SPEC 

REF 

:  MMI 

Money 

Movement  let  Authorise 
Movement  2nd  Authorise 
Movement  Amend/Del ete/Mrlte-off 
Movement  Input/Summary  Reports 


T-PLAN  TEST  DICTIONARY  AS  AT  04/01/80  VERSION  1.1 
INPUT  CHANGE  IMPACT  ANALYSIS  REPORT 

INPUT  REF  :  EIN  Foreign  Exhange  Deal  Input 


TEST  SPEC  REP  :  DEE 
TEST  SPEC  REF  :  FA1 
TEST  SPEC  REF  :  FA2 
TEST  SPEC  REP  :  FAD 
TEST  SPEC  REF  :  FIN 
TEST  SPEC  REF  :  PEN 


General  Deal  Enquiries 
Foreign  Exchange  Deal  1st  Authorisation 
Foreign  Exchange  Deal  2nd  Authorisation 
Foreign  Exchange  Deal  Amend/Delete/Write-off 
Foreign  Exchange  Deal  Input/Summary  Reports 
Position  Enquiries 


T-PLAN  TEST  DICTIONARY  AS  AT  04/01/80  VERSION  1.1 
OUTPUT  CHANGE  IMPACT  ANALYSIS  REPORT 

OUTPUT  REP  :  FIS  FX  Deal  Input  Summary  Report 


TEST  SPEC  REP  s  FIN  Foreign  Exchange  Deal  Input/Suamary  Reports 


Figure  23-10.  T-PLAN  Test  Dictionary  Change  Impact  tor  Function  MME,  Input  EIN,  Output 


FP2  Forex  Fixed  Deal  2nd  Authorisation 
FFA  Forex  Fixed  Deal  Amend/Delete/Nr ite-off 
MU.  Money  Movement  1st  Authorisation 
M12  Money  Movement  2nd  Authorisation 
MU  Money  Movement  I  nput /Summary  Reports  < 

MMA  Money  Movement  Amend/Delete/Hrlte-off 
Dll  Discounted  Items  Input Summary  Reports 

NMD  Non-Working  day 

TEST  SPECIFICATION  INDEX  AS  AT  04/01/080  VERSION  1.1 


TEST  TEST  SPEC  AUTHOR 

SPEC  NAME 

REF 


PSW  Btach  FX  SMZFT  Massaga  Generation 
PUP  Batch  Position  Update/Report 

Figure  23*11.  T-PLAN  Test  Dictionary  Test  Specification  Index 
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AS  AT  13/05/92 
TIME  OF  PRINT 
23:44:27 


SOFTWARE  BUILD  :  01 

Should  sot  the  FX  Supplementary  Screen  only  be  Presented  if 
The  Program  Spec  does  not  make  it  clear 

AREA  RAISED  BY  PGM  WHO  RAISED  BY  HHG  DATE  RAISED  01/09/91 

PRIORITY  CODE  BY  02/09/92 

SERVICE  QUERY  CONKBRS 


No  mention  ia  in  the  program  spec  regarding  the  criteria  for 
display  of  the  FX  supplementary  details  screen 
HHG  -  PGM 

Speo  amended  accordingly 
COT  -  DES 

No  action  for  STT  -  Design  spec  specifies  this 
HH  -  STT 


SERVICE  QUERY 
NUMBER 
00002 


SYSTEM  :  IBS  PHASE/PACKAGE  :  1.1 

DESCRIPTION 


SERVICE  QUERY  STATISTICS 


|  DESIGN 

0.10 

DESIGN 

0.10 

DESIGN 

0.00 

1 

j  PROG 

0.00 

PROG 

0.00 

PROG 

0.00 

1 

j  SYS  TEST 

0.00 

SYS  TEST 

0.00 

SYS  TEST 

0.00 

1 

j  OTHER 

0.00 

OTHER 

0.00 

OTHER 

0.00 

1 

|  TOTAL 

0.00 

TOTAL 

0.00 

TOTAL 

0.00 

1 

SERVICE  QUERY  ACTION  AUTHORISED  BY  FF 
SERVICE  QUERY  CLASSIFICATION  03  MAJOR  DOCUKBRAIION  ERROR 

Figure  23-12.  T-PLAN  Test  Management  Service  Query  Report  for  SQ  00002 


TEST  SPECIFICATION/SERVXCE  QUIRT  LOG 


TEST  SPEC  :  PIN  -  Foreign  Deal  Input/Summary  Reports 


1 - 

1  SQ 
|  NO. 

1 

1 

1 

Raised 

fox 

Test 

Spec  7 

Date  I 
Logged  j 

1 

j 

- SPEC  UPDATE  DETAILS - 

- Updated -  — Checked — 

Raqd  By  Data  By  Date 

? 

— REGRESSION  TEST 

Regn.  Test 

Test  By 
Reqd.7 

1 

| 00001 

- 

yes 

19/09/91  j 

) 

| 00002 

no 

01/09/91  j 

Figure  23-13.  T-PLAN  Test  Management  Test  Spec/SQ  Log  for  FIN 
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SERVICE  QUERY  FREQUENCY  t  CLEAR  UP  REPORT 


1 

Period  | 

1 

Service 

Queries 

| 

From  To  | 

Raised 

1 

|  Signed-off 

10/08/91  -  17/08/91  | 

0 

1  0 

18/08/91  -  35/08/91  j 

0 

1  0 

SERVICE  QUERY  ANALYSIS  BY  CLASSIFICATION  WITHIN  SCHEDULE 


SYSTEM 

)  PACKAGE 

1  /PHASE 

| SOFTWARE | 
j  BUILD  j 

SERVICE  QUERY  CALSSIFICATION  j 

1 

QUANTITY 

IBS 

i 

1  1-1 

1 

1 

1 

1  1 

1  1 

1  01  | 

1  1 

1 

1 

1 

03-  MAJOR  DOCUMENTATION  ERROR  | 

1 

1 

1 

f 

i  1 

1  _ 

07-  MAJOR  PROGRAMMING  ERROR  j 

r 

1 

1 

1 

1  1 

|  TOTAL  for  Software  Build  01  | 

i 

3 

1  1 

I  TOTAL  for  Faokage/Pbaae  1.1  1 

3 

TOTAL  for  Systaa  IBS 

i 

3 

|  OVERALL  TOTAL 


3 


Figure  23-14.  T-PLAN  Test  Management  Service  Query  Reports 
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OVERALL  PROGRESS  AS  AT  16/03/92 


TASK 

TASK 

[ - Totals - ] 

Percent 

CODE 

DESCRIPTION 

ORIG  ACT  OUT ST 

Complete 

PHASE/PACKAGE  1.1  SOFTWARE  BUILD  01 

DEE  General  Deal  Enquiries 

13.75 

5.00 

8.25 

37.74 

« 

• 

FA1  Foreign  Exchange  Deal  1st  Authorisa 

10.50 

5.75 

5.00 

52.38 

t 

FIN  Foreign  Exchange  Deal  Input/Sumary 

13. 7S 

11.00 

3.75 

72.73 

4 

SOFTWARE  BUILD  TOTALS 

37.50 

21.75 

17.00 

54.67 

t 

PRASE/PACKAGE  1.1  SOFTWARE  BUILD  02 

FA2  Foreign  Exchange  Deal  2nd  Authorisa 

15.75 

2.00 

13.75 

12.70 

« 

• 

SOFTWARE  BUILD  TOTALS 

15.75 

2.00 

13.75 

12.70 

« 

PHASE/PACKAGE  TOTALS 

53.25 

23.75 

30.75 

42.25 

« 

• 

OVERALL  TOTALS 

53.25 

23.75 

30.75 

42.25 

k 

Figure  23*15.  T-PLAN  Overall  Progress  for  IBS 
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Figure  23-17.  T-PLAN  Test  Management  Reports 
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Figure  22-17  continued:  T-PLAN  Test  Management  Reports 
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24.  TBGEN  and  TCMON 

TBGEN  generates  test  drivers  that  facilitate  unit  testing  and  bottom-up  integration  test¬ 
ing.  The  latest  version  of  this  tool  includes  the  generation  of  stubs  so  that  top-down  integra¬ 
tion  testing  is  also  supported.  TBGEN ’s  companion  tool,  TCMON,  provides  structural 
coverage  and  timing  analysis. 


24.1  Tool  Overview 

Until  recently,  TBGEN  and  TCMON  were  marketed  by  1CL  Personal  Systems,  former¬ 
ly  Nokia  Data  Systems.  They  are  now  available  from  Testwell  Oy.  These  tools  have  been 
commercially  available  since  1986  and  30  permanent  multi-user  licenses  have  been  sold. 
Designed  to  be  hardware  architecture,  operating  system,  and  compiler  independent,  these 
tools  are  available  for  VAX/VMS,  Sun-3/SunOS,  PCs  under  MS-DOS  and  OS-2,  and  Ra¬ 
tional  machines.  There  are  some  minor  difference  between  the  versions  available  on  differ¬ 
ent  operating  environments;  for  example,  unlike  the  Sun-3/SunOS  versions,  the  VAX/VMS 
tools  do  not  allow  escaping  to  the  operating  system  command  level.  At  the  time  of  exami¬ 
nation,  TBGEN  prices  started  at  $2,850  and  TCMON  at  $2,300.  The  versions  examined 
l  were  TBGEN  Version  3. 1  and  TCMON  Version  2.2  operating  on  a  VAX/VMS  platform. 


24.1.1  TBGEN  Overview 

►  Using  Ada  program  unit  specifications,  TBGEN  generates  a  test  driver  and  a  command 
file  for  compiling  and  linking  this  test  driver  with  the  units  under  test.  The  user  can  control 
the  size  of  the  resulting  testbed  by  specifying  particular  subprograms  or  program  objects  to 
be  excluded.  A  log  file  automatically  records  pertinent  information  about  testbed  genera- 

►  tion.  The  user  executes  the  resulting  testbed,  interactively  specifying  the  desired  calling  se¬ 
quence  and  subprogram  parameters,  and  observing  the  results.  (Since  the  testbed  takes 
standard  input  from  the  keyboard  for  interactive  communication  with  the  user,  some  diffi¬ 
culties  may  be  encountered  if  a  module  under  test  also  uses  standard  input.) 

A  powerful  set  of  Ada-like  testbed  commands  is  provided.  For  example,  testbed  vari¬ 
ables  can  be  declared  and  their  visibility  directly  controlled,  and  many  of  the  entities  de¬ 
clared  in  Ada  specifications  can  be  accessed.  Additional  commands  display  information 
based  on  current  testbed  settings  and  testbed  status,  or  cause  user  inputs  and  testbed  outputs 
to  be  copied  to  a  trace  file  for  later  examination.  Instead  of  using  a  testbed  interactively,  the 
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user  can  specify  testbed  inputs  in  the  form  of  a  script  file.  Scripts  may  be  user  developed  or 
generated  from  a  copy  of  previous  testbed  inputs.  Conditional  and  iteration  control  struc¬ 
tures,  along  with  fixed  and  variable  breakpoints,  are  provided  for  scripts.  Assertions  are 
provided  for  automatic  checking  of  test  results  against  expected  results. 


24.1.2  TCMON  Overview 

TCMON  instruments  the  contents  of  user-selected  files  with  statements  that  act  as  mea¬ 
surement  probes.  These  probes  provide  for  coverage  analysis  at  the  segment,  condition,  and 
subcondition  levels.  In  addition  to  structural  coverage,  probes  provide  for  segment  execu¬ 
tion  counts  and  true/false  counts  of  conditions  and  subconditions.  They  also  provide  timers 
that  allow  capturing  execution  times  at  the  program  unit  level  and  the  measurement  of  times 
between  user-specified  events.  Each  subprogram  can  be  instrumented  for  different  types  of 
monitoring.  A  test  monitor  is  generated.  A  command  file  for  compiling  the  monitor  and  in¬ 
strumented  code  and  performing  necessary  linking  is  also  generated,  together  with  a  log  file 
providing  information  about  instrumented  files  and  units  generated.  The  monitor  supports 
a  command-driven  interface  that  provides  the  user  with  commands  such  as  those  required 
to  reset  all  counters  and  timers,  save  and  append  measurement  data,  produce  a  profile  list¬ 
ing,  and  run  the  instrumented  program.  Where  necessary,  this  interface  can  be  omitted  by 
inserting  TCMON  commands  in  source  files  as  special  comments  and  generating  a  dummy 
monitor.  Data  generated  by  the  instrumentation  is  recorded  in  a  profile  listing.  This  gives 
detailed  information  about  counter  and  timer  places  and  values,  and  a  histogram  of  state¬ 
ment  list  execution  counts  is  included.  The  profile  listing  also  contains  information  that  can 
be  used  to  estimate  the  influence  of  instrumentation  statements  on  measured  time.  The  TC¬ 
MON  Postprocessor  (TCPOST)  processes  the  profile  listing  to  generate  summary  reports 
at  either  the  package  or  subprogram  level. 

Timers  may  include  invalid  data  when  two  or  more  tasks  call  the  same  instrumented 
subprogram  or  are  of  the  same  instrumented  task  type.  The  same  is  true  for  recursive  pro¬ 
cedures.  If  this  happens,  the  affected  timers  are  flagged  in  the  profile  listing.  Although  ge¬ 
neric  procedures  and  packages  can  be  instrumented,  multiple  instances  are  not 
distinguished.  Also,  when  returning  from  a  function,  it  is  not  possible  for  a  timer  within  the 
function  to  record  the  time  spent  in  the  evaluation  of  the  return  expression.  Exceptions, 
which  are  invisible  to  the  instrumentor,  may  also  distort  timing  results. 
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24.2  Observations 

»  Ease  of  use.  The  user  interacts  with  TBGEN  and  TCMON  through  command  interfaces 

that  are  well  supported  with  prompts  to  guide  a  user  through  necessary  steps.  Context-sen¬ 
sitive  help  is  available,  together  with  general  descriptions  on  user-selected  topics.  Error 
messages  are  informative,  though  no  specific  help  for  resolving  an  error  is  provided;  mes- 
i  sages  are  written  to  both  the  display  and  the  appropriate  log  file.  When  erroneous  input  is 

detected,  execution  of  the  current  command  is  terminated  and  the  rest  of  the  current  input 
line  ignored.  When  a  test  script  is  being  used  in  TBGEN,  processing  will  continue  with  the 
next  line.  Command  files  are  provided  to  relieve  the  user  of  some  repetitive  manual  labor. 

»  Although  the  use  of  TCMON  requires  no  special  knowledge,  the  TBGEN  command-inter- 

face  requires  some  knowledge  of  Ada.  All  reports  are  well-structured  and  clear,  with  useful 
history-keeping  information. 

TBGEN  is  tailorable  in  several  ways.  The  SPECIAL  command  implements  environ- 
’  ment  or  installation  specific  commands.  Configuration  parameters  specified  in  a  system  file 

can  be  changed,  essentially  to  modify  default  file  names.  A  system  file  gives  the  specifica¬ 
tion  for  package  STANDARD  which  can  be  modified  to  reflect  some  of  the  options  avail¬ 
able  to  Ada  compilers.  The  template  files  used  in  generating  testbed  components  can  be 
changed. 

Some  aspects  of  TCMON  can  also  be  altered  by  modifying  the  template  file  used  for 
generating  auxiliary  Ada  units  and  the  command  file.  This  template  file  also  contains  the 
(  configuration  parameters  that  can  be  changed  to  alter  default  values.  The  TCMON  User’s 

Manual  provides  suggestions  for  modifying  the  parent  type  for  counter  variables,  measur¬ 
ing  CPU  time  instead  of  default  elapsed  time,  and  including  other  cost  functions. 

Documentation  and  user  support.  The  documentation  is  well-written  and  guides  a 
user  through  using  each  tool.  The  vendor  provided  good  support  and  answered  all  questions 
quickly  and  well. 

Instrumentation  overhead.  TCMON  is  designed  to  minimize  the  introduction  of  un¬ 
necessary  instrumentation.  It  not  only  allows  the  user  to  select  the  files  whose  contents  are 
to  be  instrumented,  but  allows  each  file  to  be  instrumented  differently.  TCMON  also  allows 
the  user  to  select  between  SAFE  or  UNCHECKED  modes  for  the  segment  counter.  The 
vendor  cites  a  50%  to  100%  increase  in  code  size  for  full  structural  instrumentation.  For  the 
Ada  Lexical  Analyzer  Generator,  full  structural  instrumentation  of  all  units  gave  a  size  in¬ 
crease  of  120%. 
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Ada  restrictions.  TBGEN  accepts  any  valid  Ada  code.  Expressions,  however,  are 
skipped  with  the  result  that  the  type  of  an  array  index  cannot  always  be  determined  auto¬ 
matically  and  the  user  may  be  asked  to  supply  this  information.  Tasks,  task  types,  and  de¬ 
pendent  entities  are  ignored  and  cannot  be  accessed  in  testbeds  directly.  Similarly,  testbeds 
do  not  provide  the  user  with  access  to  objects  of  limited  type,  functions  with  results  of  lim¬ 
ited  type,  array  objects  with  a  constrained  array  definition,  and  constrained  subtypes  of  a 
type  with  discriminants.  TCMON  may  misinterpret  overloaded  operators  returning  boolean 
values  when  these  are  used  in  conditions. 

Problems  encountered.  No  significant  problems  were  encountered  during  the  exami¬ 
nations  of  these  tools. 


24.3  Recent  Changes 

TBGEN  version  3.0  has  been  ported  to  Apollo/Domain  environments.  An  upgrade,  ver¬ 
sion  3.1,  is  available  on  a  limited  set  of  platforms.  The  notable  enhancements  included  in 
the  upgrade  are  a  recording  facility  for  user  input  to  allow  automatic  repetition  of  interac¬ 
tive  test  sessions  and  a  blockwise  USE  command. 

In  November  1991,  TBGEN  version  4.0  was  released  and  is  now  available  for  all  the 
previous  environments,  except  Rational  machines.  This  version  introduces  the  generation 
of  stubs  to  facilitate  top-down  testing.  TBGEN  is  the  only  identified  tool  that  provides  this 
powerful  and  valuable  capability. 

TBGEN  and  TCMON  are  also  available  through  DDC  International  as  an  integrated 
part  of  its  CASE  Toolbox  product.  The  Sun/SPARC  version  of  these  tools  is  also  available 
through  DDC  International. 

Under  certain  circumstances  both  tools  can  be  licensed  in  Ada  source  code  form  with 
connection  ports  to  Gould,  Apollo/Domain,  and  some  NEC  machines  (Unix  environments). 

24.4  Sample  Outputs 

Figures  24-1  through  24-6  provide  sample  outputs  from  TBGEN  and  TCMON. 
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--  Script  file  :  USR : [ADATEST . TBGEN] CALENDAR . REC ; 1 
--  Created  at  :  1991-08-15  10:37:14 

—  Created  by  :  Test  bed  CAL_BED  generated  at  1991-08-15  09:00:03 
SET  TRACE  FILE  calendar. trc 


DECLARE 

USE  calendar 
moment 
current_year 
current_month 
the_day 
seconds 
BEGIN 


tine  : -  clock 

year 

month 

day_num 

day_dura 


split (moment,  current  year,  current_month, 


the_day, 


seconds ) 


moment  :«  time_of (current_year,  current_month ,  15,  0.0) 
DISPLAY  day (moment) 

moment  :-  add  1 (moment,  86400.0)  —  add _ 1  equiv  to  "+" 

split (moment ,  current_year ,  current_month ,  the_day,  seconds) 
ASSERT  the_day  ■  16  AND  seconds  -  0.0 


now  :  time  :«  clock 

later  :  time  : -  clock 

ASSERT  le _ l(now,  later)  -  true  —  le _ 1  equiv  to  *<-* 

moment  :•  time_of (1991,  2,  28,  0.0) 

ASSERT  NOT  EXCEPTION 

moment  ;«  tine_of (1991,  2,  29,  0,0) 

ASSERT  EXCEPTION (time_error) 

END 

SET  TRACE  CLOSED 
SET  RECORD  CLOSED 


Figure  24-1.  TBGEN  Record  File 
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Softplan  (R)  Ada  Tools 

TBGEN  System  Version  3.1,  Copyright  (C)  1990  Nokia  Data  Systems 
Test  Bed  Trace  Listing 


Test  bed 
_BED> 
_BED> 

~bm» 

_BED> 
~BED> 
_BED> 
_BED> 
BED> 
BED> 


_BED> 

~BED> 

BED> 


_BED> 

BED> 


_BED> 
~BED> 
_BED> 
_BED> 
_BED> 
~BED> 
_BED> 
_BED> 
CAL_BED> 
CAL_BED> 

CAL_BED> 
CAL_BED> 
CAL  BED> 


generated  at  1991-08-15  09:00:03.  Time  is  now  1991-08-15  10:37:22 
DECLARE 

USE  calendar 

moment  :  time  : ■  clock 

current_year  :  year 
current_month  :  month 
the_day  :  day_num 

seconds  :  day_dura 

BEGIN 

split (moment,  current  year ,  current_month,  the_day,  seconds) 
YEAR  (out)  -  1991 

MONTH  (out)  -  8 

DAT  (out)  -  15 

SECONDS  (out)  -  38266.8500 

moment  : -  time_of ( current_year ,  current_month ,  15,  0.0) 

DISPLAY  day (moment) 

15 

moment  : “  add  1 (moment ,  86400.0)  —  add  1  equiv  to  *+" 

split(moment ,  current_year ,  currentmonth ,  the_day,  seconds) 
TEAR  (out)  -  1991 

MONTH  (out)  *  8 

DAT  (out)  -  16 

SECONDS  (out)  -  0.0000 

ASSERT  the_day  -  16  AND  seconds  “  0.0 


now  :  time  :■  clock 
latex  :  time  : -  clock 
ASSERT  le _ l(now,  later)  •  true 


le  1  equiv  to  *  <-* 


sent  :■  time_o£(1991,  2,  28,  0.0) 
ASSERT  NOT  EXCEPTION 

moment  time_of (1991,  2,  29,  0.0) 
exception  CALENDAR . TIME_ERROR 
ASSERT  EXCEPT I ON ( t ine_er r or ) 

END 

SET  TRACE  CLOSED 

Trace  dosed  at  1991-08-15  10:43:46 


Figure  24-2.  TBGEN  Trace  File 
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Softplan  (R)  Ada  Tools 

TBGEN  System  Version  3.1,  Copyright  (C)  1990  Nokia  Data  Systems 
Test  Bed  Generation  Log  File 


Licence  identification  of  the  generator! 

Test  bed  timestamp. . . ;  1991-08-15  09:00:03 

Test  bed  name . :  CAL_BED 

Generated  Ada  files..:  cal*.ada 
Command  file . :  calCMD.COM 

Analysed  source  files: 

File:  TBGENSYS.STD 
File :  calendar . spe 


The  symbol  table  : 

package  STANDARD/ 8001/  is 
type  BOOLEAN/1/  is  ( 

FALSE, 

TRUE) ; 

type  INTEGER/2/  is  Integer_Type; 
type  FL0AT/3/NOV/  is  Float_Type; 
type  CHARACTER/4 /No V/  is  <~ 

<>); 

subtype  NATURAL/5/NOV/  is  INTEGER  <2>  ; 

—  Type  Class  ■>  Integer_Type 
subtype  POSITIVE/6/N0V/  is  INTEGER  <2>  ; 
—  Type  Class  «>  Integer_Type 

function  ■>**/2015/( 

LEFT  :  in  CALENDAR. TIME  <11>  ; 
RIGHT  :  in  CALENDAR. TIME  <11>  ) 
return  BOOLEAN  <1>  , 

—  Alias  Name  ->  GE  1 
TIME_ERROR/6 006/  :  exception; 

end  CALENDAR; 
end  STANDARD; 


Execution  of  the  generator  successfully  ended  at  1991-08-15  09:01:02 


Figure  24-3.  TBGEN  Generated  Log  File 
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*  TCMON  System  Version  2.2,  (C)  Copyright  1987  by  Softplan  * 

*  Test  Coverage  Monitor  /  Program  Bottleneck  Finder  * 

*  Execution  profile  listing  * 


Counters  Timers 

LINS  EXECUTION  LTBGEN /TCMON-  PLACE  START/  END/  AVERAGE 


NO.  COUNTS  VEL  DESCRIPTION  TRUE  FALSE  TIME  TIME 

Source  file  «>  [-.adalex2]ll_deds.ada  Instr  ->  (A,N,N,N) 

Source  file  ->  ll_ecmpile_duamy.ada  Instr  ->  (A,N,N,Y) 

23 .  1  proc  LL_COMPILE 

120 .  2  func  LLFIND 

124*****>>>>>  2  begin  198  0  ? 


127*****> >>>>>>  3  while_loop  898  763  ? 

Condition  898  63 


Source  file  ->  [-.adelex2]ll_sup_spec.ada  Instr  ■>  (A,N,N,N) 


Source 

file  ->  ll_aup_body_mt . ada 

Instr  ->  ( A,Y 

,r,Y) 

50.  .  . 

0 

0.0000 

97.  .  . 

14 

0.0000 

0.0000 

170.  .  . 

6 

0.0013 

0.0078 

175*** 

**>  2  begin 

6 

0  ? 

176 

TINER_CHAR_RANGE 

6 

0.0013 

0.0078 

178*** 

3  if_then 

3 

3 

Condition 

3 

3 

181*** 

3  if  else 

3 

3 

183*** 

••>>>>  4  for_loop 

62 

62 

188*** 

••>  3  return 

6 

0 

204.  .  . 

4 

0.0039 

0.0156 

Median  of  nonzero  counter  values  -  8 

One  asterisk  (  *  )  <■>  1 

Number  of  counters  •  539 

Number  of  timers  a  36 

Number  of  instrumented  (sub)oonditions  ■  286 

Minimum  measurable  time  interval  ■  0.0001 

Estimated  cost  of  one  timer  operation  -  0.0006 

Estimated  cost  of  one  counter  operation  -  0.0000 

The  instrumentation  was  done  1991-08-14  13 >15: 27 

This  listing  vas  produced  1991-08-14  13:32:48 

Data  files: 

NAME  • 7sampleTIM.dat 


Figure  24-4.  TCMON  Profile  Execution  Listing 
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TCMON  System  Version  2.2,  (C)  Copyright  1987  by  Softplan 
Test  Coverage  Monitor  /  Program  Bottleneck  Finder 
Log  of  TCMON  preprocessor  execution 


Date  and  time 


->  1991-08-14  13 i 15: 27 


Prefix  ->  SAMPLE 

Generated  files  ■>  sample*. ada 

Main  procedure  ->  *not  specified* 

Code  pattern  file  ->  PATTERNS. TCM 

Source  ■>  ll_sup_body_mt . ada 
Target  *■>  sample_ll_sup_body_mt.ada 
Instrumentation  ->  7  INC_M0DE  ->  UNCHECKED 

COUNTERS  ->  ALL 

AUTO_TIMERS  ->  YES 

MANUAL.TZMERS  ->  YES 

EXPAND_COMMANDS  ->  YES  ) 

package  body  LL_SUPPORT  on  line  50 
function  ALTERNATE  on  line  97 

function  NERGE_RANGES  on  line  103 
funation  CHAR_RANGE  on  line  170 
— sc  start  tiner_chax_range  on  line  176  expanded 
— it  stop  tlmer_ohar_range  on  line  187  expanded 
procedure  COMPLETEJPAT  on  line  192 

Summary  Information 


Number  of  instrumented  files 

Number  of  compilation  units 

Number  of  body  stubs 

Number  of  subunits 

Number  of  statement  list  counters 

Number  of  (sub) condition  counters 

Number  of  timers 

Number  of  manual  timer  STARTs 

Number  of  manual  timer  STOPs 

Number  of  embedded  commands 


all  expanded 
all  expanded 
all  expanded 


Command  file  for  compilation  and  linking  “>  SAMPLECMD . COM 


ERRORS:  0  WARNINGS:  0 


Figure  24-5.  TCMON  Log  File 
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*  TCMON  System  Version  2.2,  (C)  Copyright  1987  by  Softplan 

* 

*  Test  Coverage  Summary  Report 

»** 

• 

PROGRAM 

STM 

COND 

SUB 

OVER 

UNIT 

LIST 

CVRG 

COND 

ALL 

CVRG 

CVRG 

CVRG 

Source  file  «>  ll_eompile_duny .  ada 
proc  LL_COMPILE 

Instr 

->  (A,N,N,Y) 

func  LLFIND 

88 

88  - 

88  - 

88  - 

proc  LLPRTSXRING 

0 

- 

0  - 

0  - 

0  - 

proc  LLPRTTOKEN 

0 

0  - 

0  - 

0  - 

proc  LLSKIPTOKEN 

0 

- 

0  - 

proc  LLSKIPNODE 

0 

- 

0  - 

proc  LLSKIPBOTH 

0 

- 

0  - 

proc  LLFATAL 

0 

0  - 

proc  GET_CHARACTER 

0 

- 

0  - 

0  - 

0  - 

func  MAKEJTOKEN 

0 

- 

0  - 

0  - 

0  - 

proc  LLNEXTTOKEN 

100 

100 

100 

100 

proc  LLMAIN 

62 

- 

47  - 

47  - 

56  - 

body  LL_COMPILE 

100 

100 

proc  LL_COMPILE 

49 

- 

44  - 

45  - 

48  - 

Source  file  ->  [~.adalex2]ll_tokens. 

ada 

Instr 

->  (A,N,N,N) 

pack  LL_T0KENS 

proc  ADVANCE 

83 

71  - 

74  - 

79  - 

pack  LL_TOKENS 

83 

- 

71  - 

74  - 

79  - 

OVERALL  SUMMARY  : 

46 

- 

44  - 

44  - 

46  - 

NiaUr  of  partially  instrumented  or  dropped  ooaipilation  units  :  0 


This  sunary  was  generated  at  1991-08-14  13:34:48,  basad  on  the  TCMON 
execution  profile  listing  file  sample_out.dat. 

The  profile  listing  was  produced  at  1991-08-14  13: 32 <48,  and  the  actual 
TCPRE  instrumentation  was  performed  at  1991-08-14  13:15:27. 

There  were  103  places  out  of  128  where  the  coverage  percentage  was 
below  the  selected  warning' level  100  %, 


Figure  24-6.  TCMON  Coverage  Summary 
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25.  TCAT/Ada,  TCAT-PATH,  S-TCAT/Ada,  TSCOPE,  &  TDGen 

>  These  tools  are  part  of  the  Software  TestWorks  toolset  that  also  includes  SMARTS, 

CAPBAK,  and  EXDIFF  for  regression  testing.  TCAT/Ada,  TCAT-PATH,  and  S-TCAT/ 
Ada  provide  structural  coverage  analysis  at  unit  and  integration  levels.  TSCOPE  provides 
a  graphical  animation  of  the  coverage  achieved  and  software  data  visualization  relating  to 
1  software  quality,  software  performance,  and  software  capacity.  TDGen  is  a  test  data  gener¬ 

ation  tool  capable  of  generating  test  data  randomly,  sequentially,  or  using  specified  values 
based  on  a  user-defined  template. 

TRACKER  and  STATIC  are  two  additional  tools  due  for  release  in  fall  1992.  TRACK¬ 
ER  supports  problem  reporting.  STATIC,  which  currently  only  operates  on  C  code,  per¬ 
forms  static  analysis  to  look  for  such  items  as  missing  break  statements,  initialized  or 
unaccessed  arrays  and  structures,  loss  of  precision,  and  nonconformance  with  the  ANSI  C 
programming  language  standard. 


25.1  Tool  Overview 

Software  TestWorks  has  been  marketed  by  Software  Research,  Inc.for  over  five  years. 
Software  Research  also  offers  a  range  of  software  testing  services,  technical  seminars,  and 
programming  language  validation  suites.  The  tools  are  available  on  a  large  number  of  op¬ 
erating  platforms  ranging  from  PCs  to  mainframes  under  Unix,  MS-DOS,  OS-2,  and  VMS 
operating  systems.  TCAT/Ada,  TCAT-PATH,  S-TCAT/Ada,  and  TDGen  can  each  be  in¬ 
voked  via  a  command  line  or  through  a  windows-based  graphical  user  interface  in  the  OSF/ 
Motif  environment.  TSCOPE  requires  the  graphical  user  interface.  Prices  depend  on  the 
operating  environment  and,  at  the  time  of  examination,  started  at  $4,900  for  TCAT,  TCAT- 
PATH,  S-TCAT,  and  TSCOPE  together.  Over  2,500  licenses  have  been  sold  for  this  group 
of  tools.  Prices  for  TDGen  started  at  $500.  Tool  users  are  supported  by  both  a  newsletter 
and  hot-line  support. 

The  examinations  were  performed  on  a  Sun-4  copy  of  TCAT/Ada  Version  7.3,  S- 
TCAT/Ada  Version  7.6,  TCAT-PATH  Release  7,  and  TSCOPE  Release  2.  These  are  all 
recently  released  versions  and  still  subject  to  beta  testing.  The  final  tool  examined  was  TD¬ 
Gen  Release  3.2. 
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25.1.1  TCAT/ Ada  and  S-TCAT/Ada  Overview 


TCAT/Ada  provides  for  segment  or  branch  coverage  analysis  at  the  unit  level  and 
STCAT/Ada  for  call-pair  coverage  analysis  at  the  module  integration  level.  Both  of  these 
tools,  however,  operate  similarly.  The  user  starts  by  instrumenting  the  code  under  test. 
Some  control  over  the  extent  of  instrumentation  to  be  performed  is  provided  by  allowing 
the  user  to  specify  a  list  of  modules  that  are  to  be  excluded  from  instrumentation.  STCAT/ 
Ada  additionally  allows  the  user  to  provide  a  file  containing  a  list  of  function  calls  to  be 
excluded  and  a  switch  that  allows  the  user  to  specify  the  number  of  characters  in  a  function 
name  that  shall  be  treated  as  distinct.  In  addition  to  the  instrumented  code,  the  instrumentor 
yields  a  reference  listing  that  shows  segment  (or  function)  markers  and  instrumentation  sta¬ 
tistics  on  a  module-by-module  basis. 

The  instrumented  program  is  then  compiled  and  linked  with  a  provided  run-time  file. 
Software  Research  provides  different  run-time  routines  to  allow  some  flexibility  in  the  be¬ 
havior  and  performance  of  the  instrumented  program.  For  example,  standard  trace  file  pro¬ 
cessing  is  performed  without  internal  processing  or  buffering,  and  the  trace  file  is  the  full, 
unedited  trace  of  program  execution.  One  option  provides  for  in-place  data  reduction  with 
the  entire  coverage  statistics  being  accumulated  in  memory  and  the  trace  file  written  after 
the  program  exits.  Another  option  allows  the  user  to  turn  trace  sampling  on  and  off  after  a 
specified  number  of  trace  records  have  been  registered  in  memory.  A  special  multi-tasking 
run-time  routine  is  needed  to  handle  instrumented  processes  that  run  in  parallel;  in  this  case, 
a  trace  file  is  produced  for  parent  and  child  processes. 

When  running,  the  instrumented  program  queries  the  user  for  the  name  of  the  trace  file 
to  which  execution  data  will  be  written.  This  trace  file  is  subsequently  used  by  a  reporting 
utility  to  list  the  overall  coverage  achieved,  identify  hit  and  not-hit  segments,  and  produce 
histograms  showing  the  frequency  distribution  of  segment  or  function  hits,  using  either  lin¬ 
ear  or  logarithmic  scales.  All  information  is  given  in  terms  of  the  segment  (or  function) 
numbers  shown  in  the  reference  listing.  In  the  case  of  instrumented  processes  that  run  in 
parallel,  a  special  utility  is  provided  for  preprocessing  of  the  generated  trace  file(s).  This 
utility  splits  tasking  and  non-tasking  trace  records  into  several  files  so  that  a  trace  file  for 
each  task  is  created. 

The  reporting  function  can  handle  several  trace  files  at  the  same  time  and  provides  for 
cumulative  coverage  analysis  by  archiving  trace  file  information  into  an  archive  file.  With 
the  exception  of  information  on  the  sequence  in  which  segments  were  hit,  archive  files  con- 
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tain  the  same  data  as  a  trace  file.  For  cumulative  reporting,  information  on  newly  hit  or 
newly  missed  items  is  provided.  Finally,  the  user  can  restrict  coverage  reporting  to  a  spec¬ 
ified  set  of  modules,  request  reporting  on  past  coverage  using  only  named  archive  files,  and 
cause  named  modules  be  deleted  from  the  archive.  The  results  of  analysis  can  be  used  to 
control  the  extent  of  subsequent  instrumentation.  This  is  achieved  via  a  threshold  switch 
that  causes  any  module  with  percentage  coverage  greater  than  or  equal  to  this  threshold  to 
be  written  to  the  de-instrumented  file.  (The  user  can  specify  the  threshold  value,  or  use  the 
default  value  of  85%. ) 

TCAT/Ada  and  S-TCAT/Ada  both  provide  a  utility  for  creating  null  archive  files.  This 
is  used  to  ensure  that  the  coverage  reporting  covers  all  modules  in  a  program  whether  or 
not  they  have  been  executed.  This  prevents  artificially  high  initial  levels  of  coverage. 

TCAT/Ada  and  S-TCAT/Ada  also  include  an  additional  utility  that  takes  the  output  of 
the  instrumentor  to  generate,  respectively,  directed  graphs  and  call  graphs  of  the  code  under 
test  Both  of  these  graphs  are  presented  in  textual  list  form  unless  the  graphical  user  inter¬ 
face  is  used.  When  the  graphical  user  interface  is  used,  a  graphical  representation  of  the  ap¬ 
propriate  diagram  can  be  supported  by  displays  that  show,  for  example,  the  associated  code, 
path  statistics,  and  standards  limits.  In  the  case  of  S-TCAT/Ada,  the  call  graph  can  be  start¬ 
ed  from  a  user-specified  root  node  and  the  outputs  of  instrumentation  of  several  source  files 
can  be  combined  to  generate  a  call-graph  for  the  whole  program. 

25.1.2  TCAT-PATH  Overview 

TCAT-PATH  differs  from  TCAT/Ada  and  S-TCAT/Ada  in  that  coverage  reporting  ad¬ 
dresses  the  paths  executed  and  is  only  provided  on  a  single  trace  file;  archive  files  are  not 
supported.  As  with  the  two  previous  tools,  the  user  can  limit  the  amount  of  instrumentation 
performed  by  providing  a  file  containing  the  names  of  functions  not  to  instrument  TCAT- 
PATH,  however,  allows  the  user  to  further  limit  instrumentation  by  inserting  flags  in  the 
code  that  switch  instrumentation  on  and  off;  these  flags  are  given  as  a  special  type  of  com¬ 
ment  and  can  be  left  permanently  in  the  code.  The  user  also  can  specify  that  instrumentation 
of  empty  edges  be  suppressed. 

In  addition  to  the  instrumented  code  and  reference  listing,  the  instrumentor  generates  a 
separate  file  of  directed  graph  information  for  each  module.  The  same  utilities  as  are  pro¬ 
vided  with  TCAT/Ada  are  available  for  using  this  information  to  draw  directed  graphs.  The 
information  is  also  used  to  support  the  following  utilities; 
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•  apg.  This  automatic  path  generator  gives  the  complete  set  of  paths  for  a  named  mod¬ 
ule.  The  user  can  request  that  only  basis  paths  are  listed;  that  is,  those  paths  that  have 
no  iteration.  Additional  options  include  presentation  of  a  set  of  path  statistics,  speci¬ 
fication  of  the  maximum  limit  on  the  number  of  paths  to  generate,  and  specification 
of  pairs  of  segments  not  to  include  in  paths.  In  complex  cases,  apg  can  be  applied  to 
a  subgraph  instead  of  complete  directed  graphs. 

•  pathcon.  Presents  the  logical  conditions,  and  associated  predicates,  that  cause  a  path 
in  a  particular  module  to  be  executed.  It  can  be  invoked  for  a  single  path,  a  set  or  range 
of  paths,  or  all  paths. 

•  pathcover.  Presents  the  essential  paths  in  a  module,  that  is,  those  required  to  be  exe¬ 
cuted  to  achieve  100%  coverage.  Essential  paths  are  determined  based  on  the  order 
of  segment  occurrence  where  this  order  may  be  adjusted  by  sorting  path  information 
on  various  criteria.  Population  statistics  on  each  segment  are  available. 

•  cyclo.  Computes  the  cyclomatic  complexity  for  the  named  module. 

Since  these  utilities  are  invoked  for  a  single  module,  they  can  require  .nany  repetitive  op¬ 
erations  on  the  part  of  the  user.  Consequently,  two  additional  utilities,  DoPTH  and  DoCYC, 
allow  the  user  to  request  that,  respectively,  apg  and  cyclo  are  applied  to  all  appropriate 
modules.  DoPIC  provides  a  similar  facility  for  drawing  directed  graphs 

The  instrumented  program  is  compiled  and  linked  as  before  and,  again,  a  number  of 
special  run-time  routines  are  available  to  provide  some  control  over  its  behavior  and  per¬ 
formance.  When  executed,  the  instrumented  program  generates  a  trace  file  for  subsequent 
coverage  analysis.  The  basic  coverage  utility  analyzes  this  trace  file  to  report  on  the  path 
coverage  achieved  for  a  named  module.  For  each  path  in  the  module,  the  coverage  report 
specifies  whether  it  was  executed  and,  if  so,  how  many  times,  together  with  an  overall  cov¬ 
erage  value.  A  special  DoRPT  utility  invokes  the  coverage  analyzer  for  all  modules  sup¬ 
ported  by  apg-generated  path  information. 


25.13  TSCOPE  Overview 

TSCOPE  is  used  with  the  trace  files  produced  by  TCAT/Ada,  TCAT-PATH,  or  S- 
TCAT/Ada  to  animate  test  coverage.  All  TSCOPE  commands  are  treated  as  primitives  that 
are  invoked  to  present  a  variety  of  displays.  Consequently,  all  instrumented  modules  can 
be  reported  on  a  single  X-Window  screen  and  different  kinds  of  reporting  can  be  selected 
for  different  modules.  (Each  program  module  can  be  instrumented  for  either  segment,  path, 
or  call-pair  coverage;  since  these  are  incompatible,  only  one  type  of  information  can  be  re¬ 
ported  from  each  module.)  Displays  are  positioned  on  the  screen  using  a  special  configura¬ 
tion  file  or  interactively  using  TSCOPE  menu  options. 
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The  following  types  of  dynamic  displays  are  available: 

•  TScldig.  Provides  dynamic  display  of  segment  or  path  coverage  data  on  the  directed 
graph  for  a  named  module.  Five  different  animation  styles  are  available. 

•  TShisto.  Provides  a  dynamic  linear  histogram  of  segment  (or  path)  coverage  and  call- 
pair  coverage  for  a  named  module.  This  histogram  reflects  the  percentage  of  times  a 
segment  is  hit.  It  is  supported  by  data  on  the  number  of  times  a  segment  is  hit  and  the 
current  segment  coverage. 

•  TSlhisto.  Provides  a  dynamic  logarithmic  histogram  of  segment  (or  path)  coverage 
and  call-pair  coverage  for  a  named  module.  This  display  provides  similar  information 
to  the  linear  histogram  display,  except  that  it  shows  the  differences  between  relative 
segment  hit  counts  more  clearly. 

•  TSsOcg  and  TSslcg.  Provides  dynamic  display  of  coverage  data  for  a  named  module 
on  one  of  two  types  of  call  tree.  TSsOcg  presents  a  call  tree  that  shows  each  distinct 
link  between  an  invoking  and  invoked  module;  it  is  used  for  showing  coverage  with 
respect  to  the  percentage  of  modules  invoked.  TSslcg  gives  only  one  line  for  each 
invoking-invoked  relationship,  regardless  of  the  number  of  connections,  and  is  used 
for  display  of  call-pair  coverage  data. 

•  TSstrip.  Provides  a  dynamic  strip  chart  that  shows  the  accumulation  of  segment  (or 
path)  coverage  for  a  named  module  during  a  single  test.  This  chart  is  supported  by 
data  on  the  percentage  of  segments  (or  paths)  that  have  been  hit  and  the  number  of 
times  the  module  is  called. 

TSCOPE  also  supports  two  static  displays.  These  are  provided  by  the  utility  available  with 
TCAT/Ada  and  TCAT-PATH  for  graphical  display  of  directed  graphs,  and  that  available 
with  S-TCAT/Ada  for  graphical  display  of  call  trees.  A  number  of  additional  utilities  sup¬ 
port  display  management 


25. 1.4  TDGen  Overview 

TDGen  generates  test  data,  or  test  files,  from  user  defined  specifications.  It  is  particu¬ 
larly  useful  for  generating  the  large  amounts  of  test  data  needed  in  stress  testing. 

TDGen  works  with  two  files.  The  Template  File  tells  TDGen  how  to  generate  test  data 
based  on  data  supplied  in  a  Values  File.  TDGen  replaces  variable  fields,  called  descriptors, 
in  the  template  with  values  from  the  Values  File.  Descriptors  may  be  user  defined  or  take 
one  of  the  predefined  values  (ASCII,  alpha,  decimal,  and  real).  In  the  Values  File,  descrip¬ 
tors  are  associated  with  potential  values.  Special  notations  are  provided  for  specifying  rang¬ 
es  of  values  and  handling  comments,  blanks,  and  other  white  space.  Once  the  Values  and 
Template  files  have  been  created,  the  user  can  invoke  test  data  generation  in  one  of  three 
ways  to  specify  how  values  should  be  taken  for  the  descriptors  in  the  Template  File; 
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•  Specifically.  The  values  for  all  or  some  of  the  descriptors  are  specified  by  integers. 

•  Sequentially.  Values  are  taken  sequentially  from  the  Values  File  to  generate  every 
possible  combination  of  the  given  values. 

•  Randomly.  Selects  values  from  the  Values  File  randomly  by  taking  one  value  from 
each  field  in  the  file  at  random.  For  each  field  name  encountered  in  the  Template  File, 
a  uniformly  distributed  random  number  is  used  to  select  a  particular  value  from  those 
corresponding. 

Additionally,  the  user  can  request  TDGen  to  tabulate  the  number  of  possible  test  data 
combinations  that  will  be  generated  to  allow  a  review  of  the  size  of  the  results  before  gen¬ 
eration  commences. 


25.2  Observations 

Ease  of  use.  A  user  can  interact  with  these  tools  using  either  a  command-line  interface 
or  a  series  of  menus.  Context-sensitive  help  and  help  frames  discussing  each  function  are 
provided.  No  special  knowledge  is  required  to  use  these  tools. 

TCAT/Ada,  S-TCAT/Ada,  and  TCAT-PATH  all  provide  a  number  of  utilities  that  can 
be  invoked  for  individual  program  units.  In  most  cases,  a  special  utility  is  available  to  apply 
the  utility  for  all  available  program  units  using  a  single  command.  Additionally,  TCAT- 
PATH  supports  Unix-like  make  files  to  facilitate  repetitive  compilation  and  linking  tasks. 

A  limited  amount  of  tailoring  is  provided  by  the  use  of  configuration  files.  These  allow 
the  user  to  adjust,  for  example,  setting  the  maximum  number  of  nodes  to  process,  format¬ 
ting  options  for  diagraph  display,  and  default  path  names. 

Reports  are  generally  well-structured.  Since  segments,  paths,  and  call-pairs  are  referred 
to  by  number,  however,  a  user  must  refer  back  to  the  various  reference  listings  to  identify 
the  subject  of  each  reference. 

Documentation  and  user  support.  The  tools  are  supported  by  extensive  documenta¬ 
tion  that  includes  guidelines  on  appropriate  minimum  coverage  levels. 

Instrumentation  overhead.  TCAT/Ada,  TCAT-PATH,  and  S-TCAT/Ada  instrument 
the  contents  of  files  specified  as  part  of  the  tool  invocation.  In  each  case,  all  code  is  instru¬ 
mented  the  same  way.  For  TCAT/Ada,  the  vendor  recommends  a  capacity  of  up  to  2,500 
segments  (approximately  20,000  lines  of  code).  The  vendor  estimates  the  size  or  perfor¬ 
mance  overhead  for  instrumentation  at  20%  to  30%,  although  this  can  be  higher  for  very 
complex  programs.  For  the  Ada  Lexical  Analyzer  Generator,  TCAT/Ada,  TCAT-PATH, 
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and  S-TCAT/Ada  instrumentation  gave,  respectively,  37%,  37%,  and  28%  increases  in 
source  code  size  with  corresponding  increases  of  15%,  12%,  and  15%  for  executable  code. 
*  Versions  of  TCAT  and  S-TCAT  that  accomplish  various  levels  of  in-place  buffering  to  en¬ 

hance  performance  are  available  for  C  programs.  Similar  support  is  not  available  for  the 
Ada  versions. 

^  Ada  restrictions.  The  TCAT/Ada  and  S-TCAT/Ada  instrumentors  have  been  validated 

against  the  Ada  validation  suite,  a  set  of  programs  that  test  compliance  with  the  Ada  stan¬ 
dard. 

TCAT/Ada  and  S-TCAT/Ada  do  not  support  conditional  expressions  in  Ada;  such  ex- 
»  pressions  must  be  expanded  to  the  explicit  if-then-else  form.  TCAT-PATH  does  not  handle 

multiple  instances  of  a  task  or  exception  handling.  Variant  records,  compound  conditions, 
and  the  terminate  alternative  of  a  selective  wait  are  not  instrumented  by  any  of  these  tools. 

Problems  encountered.  Problems  encountered  with  the  runtime  files  and  installation 
1  instructions  of  earlier  releases  of  these  tools  have  been  fixed.  For  each  of  TCAT/Ada,  S- 

TCAT/Ada,  and  TCAT-PATH,  incorrectly  inserted  instrumentation  statements  prevented 
compilation.  In  most  cases,  however,  these  errors  were  relatively  easy  to  correct  manually. 
The  TCAT-PATH  pathcon  utility  gave  a  segmentation  fault  after  processing  the  first  record 
1  of  generated  trace  files,  and  the  coverage  reporting  utility  could  not  report  the  coverage 

achieved  for  some  program  units;  TS  strip  was  the  only  utility  that  consistently  worked  as 
expected,  though  TScldig  worked  as  long  as  certain  command  options  were  not  given.  It 
was  not  possible  to  get  several  windows  displayed  on  a  screen.  Software  Research  are  in- 
!  vestigating  these  problems.  In  TDGen,  errors  in  values  and  template  file,  or  in  the  specifi¬ 

cation  of  program  options,  caused  the  program  to  hang. 

,  25.3  Recent  Changes 

Software  TestWorks  has  been  integrated  with  IBM’s  AIX  Software  Development  En¬ 
vironment  Workbench/6000. 


25.4  Sample  Outputs 

Figures  25-1  through  25-26  provide  sample  outputs  from  these  tools. 

1 
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TCAT  Series  &  TDGen 


PART 


—  TCAT/Ada,  Release  2.1  for  SUN  (09/16/92). 

—  (c)  Copyright  1989  by  Software  Research,  Inc.  ALL  RIGHTS  RESERVED 

—  SEGMENT  REFERBfCE  LISTING  Erl  Sep  25  13:26:32  1992 

procedure  LLFATAL  Is 

—  To  recover  from  syntactic  error,  terminate  compilation 
begin 

— •  Module  nev_ll_compile . LLFATAL  » — 

— *  Segment  1  <>  » — 

PUT(  STANDARD_ERROR ,  ■***  Fatal  *  ) ; 

LLPRTTOKEN; 

PUT(  STANDARD_ERROR,  "  found  in  line  *); 

PUT(  STAND ARD_ERR0R ,  LLCURTOK . LINENUMBER ,  1  ); 

PUT_LINE(  STAND ARD_KRR0R ,  •  —  terminating  translation."  ); 
raise  PARSING_£RKOR ; 
end  LLFATAL; 

— *  End  module  new_ll_conpile . LLFATAL  • — 

begin  —  TESTSYNCH 
— «  Module  new_ll_eompile . TESTSYNCH  * — 

— *  Segment  1  <>  * — 

while  LLSTACK ( LLSENTPTR ) . DATA . STNCHINDEX  -  0  loop 
— *  Segment  2  <>  • — 

—  no  synch  info  there 

if  LLSTACX< LLSENTPTR) .PARENT  /«  0  then 
— •  Segment  30  * — 

—  there  really  is  a  parent 
LLSENTPTR  i-  LLSTACK (LLSENTPTR) .PARENT; 

else 

— *  Segment  4  <>  * — 

LLFATAL; 
end  if; 
end  loop; 

— •  Segment  5  <>  • — 

SYNCHRONIZE; 
end  TESTSYNCH; 

— *  End  module  new_ll_ocmpile . TESTSYNCH  • — 

begin  —  LL_CONPILE 
— •  Module  oew_ll_caapile . LL_COMP  ILE  » — 

— •  Segment  1  <>  • — 

OPEN(  LLSOURCE,  INJKLE,  "SOURCE"  ); 

LLMAIN ; 

CLOSE (  LLSOURCE  ); 
end  LL_COMPXLE; 

—  BID  OF  TCAT/Ada  REFERENCE  LISTING 


Figure  25-1.  TCAT/Ada  Reference  Listing  for  LL_COMPILE 
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TCAT/Ada ,  Release  2.1  for  SUN  (09/16/92). 


INSTRUMENTATION  STATISTICS 


Module 


#  segments  t  statement*  I  Conditional  stataaents 


new_U_oc*pile . LLPIN 
nev_ll_co*pile . LLPRT 
new_ll_oo*pile .  LIiPRT 
new_ll_oo«pile . LLSKI 
nev_ll_ooMpile . LLSKI 
n«w_ll_ooBpile .  LLSKI 
n«v_ll_eoapile .  LLFAT 
netr_ll_ccapile . GET_C 
newllooapile . CVT_S 
n«w_ll_co«pile ,MAKE_ 
nev_ll_oo«p il e . LLNEX 
new_ll_coap 11 e . BUILD 
n«w_ll_ooaplle . BUILD 
n#w_ll_oo«pile . READG 
n«w_ll_ooapile . ERASE 
new_ll_ocapile .MATCH 
netr~ll_aaapile .  expan 
new_ll_co«pile . SYNCH 
nev_ll_ca«pile . TESTS 
nee~ll~oo«pile . PARSE 
n«v_ll_ca«pile . LLMAI 
new_ll_eoapile . LL_CO 


e 

11 

3 

5 

5 

2 

3 

4 

1 

l 

7 

0 

l 

8 

0 

l 

9 

0 

l 

6 

0 

4 

7 

1 

5 

5 

2 

15 

18 

4 

3 

5 

1 

15 

32 

5 

3 

5 

1 

11 

24 

5 

5 

7 

2 

7 

6 

3 

13 

21 

6 

19 

29 

9 

5 

5 

2 

18 

29 

6 

1 

2 

0 

1 

3 

0 

Figure  25-2.  TCAT/Ada  Instrumentation  Statistics  for  LL_COMPILE 
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Figure  25-3.  TCAT/Ada  Directed  Graph  for  LLFIND  from  LL_COMPILE 
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TCAT  Series  &  TDGen 


Coverage  Analyser.  (Release  8.3  for  SUN/UNIX  09/16/931 
(c)  Copyright  1990  by  Software  Research,  Inc. 

Selected  COVER  System  Option  Settings: 


I-C) 

Cumulative  Report 

—  NO 

(-pi 

Past  History  Report 

—  NO 

l-nj 

Not  Hit  Report 

—  yes 

t-H) 

Hit  Report 

—  YES 

l-nhl 

Newly  Hit  Report 

—  NO 

t-nmj 

Newly  Missed  Report 

—  NO 

t-hl 

Histogram  Report 

—  YES 

(-11 

Log  Soale  Histogram 

—  YES 

(-81 

Reference  Listing  Cl 

—  NO 

Options  read:  4 

TCAT/C:  Coverage  Analyser.  (Release  8.3  for  SUN/UNIX  09/16/93) 
(c)  Copyright  1990  by  Software  Research,  Inc. 

Cl  Segment  Hit  Report. 


No.  Nodule  Name:  Segment  Coverage  Status: 


1 

new_ll_oomplle . LL_C0MFXLE 

All 

Segments  Hit. 

Cl  - 

100* 

3 

new  JLl_pompile . LLMAXN 

All 

Segments  Hit. 

Cl  - 

100* 

3 

new'll  compile. READGRAN 

All 

Segments  Hit. 

Cl  - 

100* 

4 

new_ll_compile . BUILD RIGHT 

1 

3 

3 

4 

5 

6 

7 

8 

10 

11 

13 

13 

15 

5 

new_ll_eompile . BUILDSELECT 

All 

Segments  Hit. 

Cl  - 

100* 

6 

n«v_ll_cowpile . PARSE 

1 

3 

3 

4 

10 

11 

13 

14 

15 

16 

18 

7 

new_ll_oampile . LLFZMD 

1 

3 

3 

4 

5 

7 

8 

8 

new_ll_campile . LLNEXTTOKEN 

All 

Segments  Hit. 

Cl  - 

100* 

9 

new_ll_tokens . ADVANCE 

1 

3 

3 

4 

5 

6 

7 

8 

10 

new_ll_tokens . SCAN_PATTXRN 

1 

3 

3 

4 

5 

6 

7 

8 

9 

37 

38 

39 

40 

43 

45 

47 

49 

61 

63 

63 

64 

65 

66 

67 

68 

69 

70 

71 

74 

75 

76 

84 

88 

89 

90 

91 

98 

99 

104 

105 

106 

11 

nev_ll_ocmpile .  GET_CHARACTEE 

All 

Segments  Hit. 

Cl  - 

100* 

43 

ll_sup_body .  BOT_CHAR 

1 

10 

43 

ll_sup_body .  BaT_PATTERN_MATCH 

1 

9 

10 

13 

13 

17 

18 

30 

33 

34 

35 

37 

38 

39 

44 

ll_sup_body .  BHT_CQNCAT_RIGHT 

1 

5 

45 

ll_sup_body . EKZT_CONCAT~CASES 

1 

3 

3 

5 

7 

Number  of  Segments  Hit: 
Total  Number  of  Segments: 
Cl  Coverage  Value: 


308 

539 

58.33* 


Figure  25-4.  TCAT/Ada  Segment  Coverage  Report  using  testl.lex 
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TCAT  Series  &  TDGen 


PART  II 


TCAT/C ■  Coverage  Analyzer-  [Release  8.2  for  SlIN/UNIX  09/18/92) 
(c)  Copyright  1990  hy  Software  Research,  Inc. 

Cl  Segment  Not  Bit  Report. 


NO. 

Module  Name ■  Sagaant  Coverage 

Status: 

1 

nev_ll_ccapile . LL_C0MPILE 

All 

Segments 

Hit. 

Cl  - 

1008 

2 

nevlllocapile .  LUtAIN 

All 

Segments 

Bit. 

Cl  - 

1008 

3 

nev_ll_ooapile . READGRAM 

All 

Segments 

Bit. 

Cl  - 

1008 

4 

n«w~ll~coaplla . BUILDRIGHT 

9 

14 

3 

naw_ll_ooopila .  BUILDS ELECT 

All 

Segaents 

Bit. 

Cl  - 

1008 

17 

6 

naw_ll_ooaplla . PARSE 

5 

6 

7 

8 

9 

13 

7 

naw~ll_oaaplla . LIPINS 

6  . 

8 

nav_ll_coaplla . LLNEXTTOKEN 

All 

Segaents 

Hit. 

Cl  - 

1008 

9 

nev_ll_tokens , ADVANCE 

9 

10 

11 

10 

nav_ll_tokane . SCAN_PATTERN 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 

21 

22 

23 

24 

25 

26 

27 

28 

29 

30 

31 

32 

33 

34 

35 

36 

41 

42 

44 

46 

48 

50 

SI 

52 

53 

54 

55 

56 

57 

58 

59 

60 

72 

73 

77 

78 

79 

80 

81 

82 

83 

85 

86 

87 

92 

93 

94 

95 

96 

97 

100 

101 

102 

103 

107  108 

109 

11 

nevllooaipila .  GET_CBARACTER 

All 

] 

ft 

• 

Bit. 

Cl  - 

1008 

40 

ll_eup_body .  *WT_SCAN_PROC 

2 

41 

ll_eup_body .  BOT_SCAN_SELECT 

IS 

17 

42 

ll~«up_body . EMIT_CHAR 

2 

3 

4 

5 

6 

7 

8 

9 

11 

12 

43 

U_*up_body .  MT_PA7TERN JKATCH 

2 

3 

4 

5 

6 

7 

B 

11 

14 

13 

IS 

19 

21 

23 

26 

30 

31 

32 

44 

ll_«up_body .  B4XT_C0NCAT_RXGBT 

2 

3 

4 

45 

U_aup_body.  Btt»_CONCAT_CASES 

4 

6 

8 

Number  of  Sagaante  Mot  Bitt  221 

Votal  Number  of  Sagaante :  329 

Cl  Coverage  Value:  38.228 


Figure  25-4  continued:  TCAT/Ada  Segment  Coverage  Report  using  testl.lex 
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TCAT  Series  &  TDGen 


TCAT/C:  Coverage  Analyzer.  [Release  8.2  for  SUN/UNIX  09/16/92] 
(o)  Copyright  1990  by  Software  Research,  Zno. 

Segment  Level  Histogram  for  Module:  nev_U_compile.LL_COMPILE 


+ - + 

|  Number  of  Executions,  Normalized  to  Maximum 
|  (Maximum  -  1  Hits)  X  -  One  Hit 

j  (Scale:  100.000  Each  X  -  0.020  Hits) 

Segment  Number  Of  | 

Number  Executions  - 20 - 40— - 60---- - 60 - 100 


Average  Hits  per  Executed  Segment;  1.0000 
Cl  Value  for  this  Module:  100.0000 


TCAT/C ■  Coverage  Analyzer.  [Release  8.2  for  SUN /UNIX  09/16/92] 
(o)  Copyright  1990  by  Software  Research,  Zno. 

Segment  Level  Histogram  for  Nodule:  new_ll_ooapila . BUZLDRZGBT 


|  Number  of  Executions ,  Normalized  to  Maximum 
j  (Maximum  -  174  Hits)  X  -  One  Hit 

|  (Scale:  0.575  Each  X  -  3.480  Hits) 

Segment  Number  Of  | 

Number  Executions  >-l - 20 - 40 - 60 - 80 - 100 


(•  -  Zero  Hits) 

Average  Hits  per  Executed  Segment:  76.7692 

Cl  Value  for  this  Module:  06.6607 


Figure  25-4  continued:  TCAT /Ada  Segment  Coverage  Report  using  testl.lex 
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PART  II 


Coverage  Analyzer.  [Release  8.2  for  SUN/UNIX  09/16/92] 

(o)  Copyright  1990  by  Software  Research,  Inc. 

Selected  COVER  System  Option  Settings: 

[-cl  Cumulative  Report  —  YES 
£ — p]  Past  Bistory  Report  —  NO 
[-n]  Not  Hit  Report  —  YES 

[-HJ  Hit  Report  —  YES 

[-nh]  Newly  Hit  Report  —  YES 
(-nm)  Newly  Missed  Report  —  YES 
[-h]  Histogram  Report  —  NO 
[-11  Log  Scale  Histogram  —  YES 
[-S]  Reference  Listing  Cl  —  HO 
Options  read:  6 

TCAT/Ci  Coverage  Analyzer.  [Release  8.2  for  SUN/UNIX  09/16/92] 

(e)  Copyright  1990  by  Software  Research,  Inc. 

■i - - ■> - 1 - f 

|  |  Current  Teat  I  Cumulative  Summary  | 


1 

Nodule  Number  Of  | 

Name:  Segments:  | 

No.  Of 
Invokes 

No.  Of  ( 

Segments  Cl*  j 
Hit  Cover  j 

NO.  Of 
Invokes 

No.  Of 

Segments  Cl* 
Bit  Cover 

new_ll_ocmpile . LL_C0M 

1  | 

1 

1 

100. 

00  | 

2 

1 

100.00 

new_ll_ccmpile ,  LLMAIN 

1  | 

1 

1 

100. 

00  j 

2 

1 

100.00 

new_ll_oompile . READGR 

11  1 

1 

11 

100. 

00  j 

2 

11 

100.00 

new_ll_compile . BUI  LOR 

IS  j 

64 

13 

86. 

67  | 

128 

13 

86.67 

new_ll_ccmpile . BUILDS 

3  | 

64 

3 

100. 

00  j 

128 

3 

100.00 

new_l  l_compile .  PARSE 

1*  1 

1 

11 

61. 

11  | 

2 

11 

61.11 

nanr_ll_camplle .  LLP  INC 

8  1 

312 

8 

100. 

00  j 

510 

8 

100.00 

new_U_oompile .  LLNEXT 

3  | 

221 

3 

100. 

00  j 

355 

3 

100.00 

new_U_tokens .  ADVANCE 

11  1 

221 

9 

81. 

82  j 

355 

9 

81.82 

new_ll_tokens . SCAN_PA 

109  j 

455 

46 

42. 

20  | 

712 

52 

47.71 

new_ll_ccmpile .  GET_CH 

4  j 

1385 

4 

100. 

00  j 

2250 

4 

100.00 

new_ll_tokens . CHAR_AD 

s  1 

1238 

3 

60. 

00  j 

2018 

3 

60.00 

new_ll_tokens .  CURRBR 

1  j 

220 

1 

100. 

00  j 

353 

1 

100.00 

new_ll_ccmpile .  MAXEJZ 

15  i 

220 

13 

86. 

67  j 

353 

13 

86.67 

new_ll_oompile . CVT_ST 

5  j 

220 

5 

100. 

00  j 

353 

5 

100.00 

new_ll_ccmpile . EXPAND 

13  i 

461 

11 

84. 

62  j 

715 

11 

84.62 

n«w_ll_compila .  MATCH 

7  j 

461 

5 

71. 

43  i 

715 

5 

71.43 

new_ll_oampile . ERASE 

5  1 

707 

5 

100. 

00  j 

1105 

5 

100.00 

new_ll_tokens . LOOK_AH 

5  | 

84 

3 

60. 

00  j 

123 

3 

60.00 

11  notions . LLTAXEACTI 

1 

429 

36 

52. 

17  | 

659 

36 

52.17 

ll_sup_body .  TAIL 

18  | 

2 

4 

22. 

22  | 

2 

4 

22.22 

ll_sup  Jbody . EMZT_ALT_ 

7  j 

12 

7 

100. 

00  j 

12 

7 

100.00 

Totals 

614  | 

7569 

384 

62. 

54  | 

11884 

391 

63.68 

Figure  25-5.  TCAT/Ada  Segment  Coverage  Report  using  testl.lex  &  sample.tex 
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TCAT  Series  &  TDGen 


(c)  Copyright  1990  by  Software  Research,  tne. 

Cl  Segment  Hit  Report. 

No.  Module  Name;  Segment  Coverage  Status: 


1 

nsw_ll_ecmplle . LL_C0MPIL£ 

All 

Segments 

Hit. 

Cl  - 

loot 

2 

new~ll_oomplle . LLNA2N 

All 

Segments 

Hit. 

Cl  - 

1004 

3 

nev~ll~ocmplla .  BXADGRAM 

All 

Segments 

Hit. 

Cl  - 

loot 

4 

n  ev_l l_oomp  11  a . BUILDRIGHT 

1 

2 

3 

4 

5 

6 

7 

8 

10 

11. 

12 

13 

15 

5 

new_ll_compile . BUILD  SELECT 

All 

Segments 

Hit. 

Cl  - 

loot 

6 

new_ll_eompile . PARSE 

1 

2 

3 

4 

10 

11 

12 

14 

15 

16  . 

IB 

7 

new_ll_oompile . LLPIND 

All 

Segments 

Hit. 

Cl  - 

loot 

8 

new  U_compile . LLNZXTT0XZN 

All 

Segments 

Hit. 

Cl  - 

loot 

9 

nev~ll~tokens .  ADVANCE 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

new~ll~tokens . SCAN_PATTERN 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

16 

17 

28 

32 

37 

38 

39 

40 

43 

45 

47 

49 

61 

62 

63 

64 

65 

66 

67 

68 

69 

70 

71 

74 

75 

76 

84 

88 

89 

90 

91 

98 

99 

104 

105 

106 

107 

11 

new  llcompile . GET_CHARACTER 

All 

Segments 

Bit. 

Cl  - 

loot 

12 

newlltokens . CHAB~ ADVANCE 

1 

3 

5 

13 

new_ll_tokens . CURRINTSTMBOL 

All 

Segments 

Hit. 

Cl  - 

loot 

50 

ll_SUp_body . TAIL 

1 

5 

14 

16 

51 

ll2»up_body . EMIT_ALT_CASES 

All 

Segments 

Hit. 

Cl  - 

loot 

Number  of  Segments  Hit: 
Total  Number  of  Segments: 

Cl  Coverage  Value: 


391 

614 

63.681 


T CAT/C i  Coverage  Analyzer.  (Release  8.2  for  SUN/ONIX  09/16/92] 
(c)  Copyright  1990  by  Software  Research,  Ino. 


Cl  Segment  Newly  Bit  Report. 


No. 

Module  Name:  Segment  Coverage  Status: 

7 

new_ll_oompile . LLPIND 

6 

9 

nev~ll_tokens . ADVANCE 

9 

10 

new_ll_tokens . SCAN_PAOTRN 

10 

11 

12  13  14  15  16 

32 

107 

50 

ll_sup_body . TAIL 

1 

5 

14  16 

51 

ll~SUp_bOdy . EMIT_ALT_CASRS 

1 

2 

3  4  5  6  7 

Figure  25-5  continued:  TCAT/Ada  Segment  Coverage  Report  using  testl.lex  &  sample.lex 
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TCAT /C:  Coverage  Analyzer.  [Release  8.2  for  SUN/UNIX  09/16/32) 
(c)  Copyright  1990  by  Software  Research)  Inc. 

Cl  Segment  Mot  Hit  Report. 


No.  Module  Name:  Segment  Coverage  Status: 


1 

new_ll_ccmpile . LL_C0Npile 

All  Segments 

Hit. 

Cl  - 

loot 

2 

newllcomplle . LLMAIN 

All  Segments 

Hit. 

Cl  - 

100% 

3 

naw_ll_cc*pil# . READGRAM 

All 

Segments 

Hit. 

Cl  - 

loot 

4 

nsw~ll_complle . BUILDRIGHT 

9 

14 

5 

new_ll_ccmplle . BUILDSELECT 

All  Segments 

Hit. 

Cl  - 

100% 

6 

new_ll_compile . PARSE 

5 

6  7 

8 

9 

13 

17 

7 

new_ll_compile .  LUTZ ND 

All . Segments 

Hit. 

Cl  - 

100% 

8 

new~ll_complle .  IiNEXTTOKEN 

All 

Segments 

Hit. 

Cl  - 

100% 

9 

new_ll_tokens . ADVANCE 

10 

11 

10 

new_ll_tokens . SCAN_PATTERN 

18 

19  20 

21 

22 

23 

24 

25 

26 

27 

29  30 

31 

33 

34 

35 

36 

41 

42 

44  46 

48 

50 

51 

52 

53 

54 

55 

56  57 

58 

59 

60 

72 

73 

77 

78 

79  80 

81 

82 

83 

85 

86 

87 

92 

93  94 

95 

96 

97 

100 

101 

102 

103 

108  109 

11 

n«w_ll_coeipile .  CET_CHARACTER 

All  Segments 

Hit. 

Cl  - 

loot 

12 

new~ll~tokens . CBAR_ADVANCE 

2 

4 

13 

nsv_ll_tokann . CURHEMT_SYMBOL 

All 

Segments 

Hit. 

Cl  - 

1001 

14 

new_ll_complle . MAXEJXOKEN 

10 

15 

48 

ll_sup_body . RESQLVE_AMBIGUITY 

4 

5  8 

9 

10 

11 

12 

13 

14 

15 

16  17 

18 

19 

49 

ll_sup_body . RESTRICT 

4 

7  13 

14 

15 

16 

22 

SO 

ll_sup_body . TAIL 

2 

3  4 

6 

7 

8 

9 

10 

11 

12 

13  15 

17 

18 

51 

ll_eup_body . EMIT_ALT_CASES 

All 

Segments 

Bit. 

Cl  - 

100% 

Humber  of  Segments  Hot  Hit :  223 

Total  Number  of  Segments: 

(14 

Cl  Coverage  Value:  <3. 

(8% 

TCAT/C:  Coverage  Analyser.  [Release 

8.2  for  SON/UNIX  09/16/92] 

(o) 

Copyright  1990  by  Software  Research,  Inc. 

Cl  Segment  Newly  Missed  Report. 

NO. 

Module  Name:  Segment  Coverage  Status: 

10 

new_ll_tokena . SCAN_FATTEBN 

66  67  68  i 

59  70  71 

40 

ll_sup_body .  BOT_SCAN_PROC 

6 

Figure  25-5  continued:  TCAT/Ada  Segment  Coverage  Report  using  test1.lex  &  sample.lex 
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TCAT/C :  Coverage  Analyzer.  [Release  8.2  for  SUN/ONIX  08/16/92] 

(o)  Copyright  1990  by  Software  Research,  Inc. 

Segment  Level  Histogram  for  Module:  nev_ll_ccmpile . LL_COMPILE 

+ - - - - - f 

I  Logarithm  of  Executions,  Normalized  to  Maximum 
I  (Maximum  •  2  Hits) 

Segment  Number  Of  | 

Number  Executions  > - 1 - - - 10 - 20 - 30 - 40 — 80-100 

+ - - - - 


Average  Hits  per  Executed  Segment:  2.0000 

Cl  Value  for  this  Module:  100.0000 


TCAT/C:  Coverage  Analyzer.  [Release  8.2  for  SUN/UNIX  09/16/92] 

(c)  Copyright  1990  by  Software  Research,  Inc. 

Segment  Level  Histogram  for  Module:  nev__ll_complle . READGRAM 

: - - — — — — —  ■  - - 

|  Logarithm  of  Executions,  Normalised  to  Maximum 
j  (Maximum  -  1280  Hits) 

Segment  Number  Of  | 

Number  Executions  > - 1— — 10 - 20 - 30 - 40 — 80-100 

•1 - _______ - - :■ - - t. 
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Average  Hits  per  Executed  Segment:  150.9091 
Cl  Value  for  this  Module:  100.0000 


Figure  25-5  continued:  TCAT/Ada  Segment  Coverage  Report  using  testl  .lex  &  sample.tex 
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--  TCAT-PATH/Ada ,  Release  2.1  for  SUN  (09/18/92). 

—  (o)  Copyright  1989  by  Software  Research,  Inc.  AL  RIGHTS  RESERVED. 

—  SEGMENT  REFERENCE  LISTING  Mon  Oct  2ti  13:09:12  1992 
procedure  LLNEXTTOKEN; 

—  get  the  next  token  from  the  input  stream  (defined  below) 

function  LLFIND(  ITEM:  LLSTRINGS ;  WHICH :  LL STYLE  )  return  INTEGER  is 
—  Find  item  in  symbol  table  —  return  index  or  0  if  not  found. 

—  Assumes  symbol  table  is  sorted  in  ascending  order. 

LOW,  MIDPOINT,  HIGH:  INTEGER; 
begin 

— •  Module  new_ll_compile.LLFIND  • — 

— •  DIGRAPH  NODE  1  * — 

— *  Segment  1  <>  * — 

LOW  :-  1; 

HIGH  :•  LLTABLESIEE  +  1; 

— *  DIGRAPH  NODE  2  * — 

while  LOW  /-  HIGH  loop 
*  — *  Segment  2  0  * — 

MIDPOINT  :-  (HIGH  +  LOW)  /  2; 

— *  DIGRAPH  NODE  3  • — 

if  ITEM  <  LLSYMBOLTABLE( MIDPOINT)  .KEY  then 
— «  Segment  3  <>  * — 

HIGH  :«  MIDPOINT; 

elsif  ITEM  -  LLSYMBOLTABLE ( MIDPOINT ) . KEY  then 
— *  Segment  4  <>  • — 

— «  DIGRAPH  NODE  4  *— 

if  LLSYMBOLTABLE (MIDPOINT) .KIND  -  WHICH  then 
— *  Segment  5  <>  * — 

return (  MIDPOINT  ); 

else 

— •  Segment  6  <>  • — 

return (  0  ); 
end  if; 

else  —  ITEM  >  LLSYMBOLTABLE (MIDPOINT) .KEY 
— *  Segment  7  <>  * — 

LOW  i-  MIDPOINT  +  1; 
end  if; 
end  loop; 

— *  Segment  8  <>  * — 

return (  0  );  —  item  is  not  in  table 

end  LLFIND ; 

— *  DIGRAPH  NODE  5  *— 

— *  End  module  nev_ll_campile . LLFIND  * — 


—  END  OF  TCAT-PATH/Ada  REFERENCE  LISTING 


Figure  25-6.  TCAT-PATH  Segment  and  Node  Reference  Listing  for  LL_COMPILE 
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—  TCAT-PATH/Ada ,  Ralaaaa  2.1  for  SUN  (09/16/92). 

—  INSTRUMENT  AT  ION  STATISTICS 


—  Nodule  1 

)  segments 

1  statements 

I  Conditional  statements 

—  naw_ll_coapile . LLFIN 

8 

11 

3 

—  naw_lI_co*iplla ,  LLPRT 

5 

5 

2 

—  nav_l l_co»pl la . LLPRT 

3 

4 

1 

—  nav_ll_coapile . LLSKI 

1 

7 

0 

—  new_ll_co«pile . LLSKI 

1 

8 

0 

—  naw_ll_ooaipila .  LLSKI 

1 

9 

0 

—  naar_ll_cc«pila . LLFAT 

1 

6 

0 

—  naw_ll_ooapile . GET_C 

4 

7 

1 

—  nev_ll_campile .  CVT_S 

5 

5 

2 

—  naw_ll_co«pile . NAKE_ 

15 

18 

4 

—  naw_ll_coapile.LLNEX 

3 

5 

1 

—  nav_ll_aoapile . BUILD 

15 

32 

5 

—  nev_ll_co«pile . BUILD 

3 

5 

1 

—  nav_ll_coapila . READG 

11 

24 

5 

—  naw_ll_co*pila . ERASE 

5 

7 

2 

—  naw_ll_co»pl la . MATCH 

7 

6 

3 

—  new_ll_co*pile . EXPAN 

13 

21 

6 

—  n«v_ll_coapila . SYNCH 

19 

29 

9 

—  new_ll_co«pile . TESTS 

5 

5 

2 

—  naw_ll_ooaipil* .  PARSE 

18 

29 

6 

—  naw_ll_ooapila .  LU4AI 

1 

2 

0 

—  nevllooaplle . LL_CO 

1 

3 

0 

Figure  25-7.  TCAT- 

PATH  Instrumentation  Statistics  for  LLFIND 

cydo  [Ralaaaa  3.3  —  9/26/90] 

Cycleaatie  Number  -  Edge*  -  Nodaa  +  2  -  8-5  +  2  -  5 

Figure  25-8.  TCAT-PATH  Cyclomat Ic  Complexity  of  Function  LLFIND 
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n«w_ll_coapile . LLFIND  8 
nev~ll~coapile .  LLPRTSTRING 
n«w_ll_co»pil« . LLPRTTOKEN 
new_ll_cc®pile . LLSKIPTOKEN 
naw_ll~co«pile .  LLSKIPNODE 
new_ll_co«»pil« .  LLSKIPBOTH 
now_ll_co*pi  la .  LLFATAL  1 
aew_ll_co*pile . GET_CHARACTER 
new_ll_ccaiplle .  CVT_STRING 
n«w_ll_compile . MAKEJTOKEN 
D«v_ll~compil« . LLNEXTTOKEN 
D«v_ll_coaplle . BUILDRIGHT 
n«w_ll_co»pila . BUILD SELECT 
n«w_ll_caMpila . READGRAM  11 
n«v_ll_caapile . ERASE  5 

n»w_ll_co«pil« . MATCH  7 

nowllcoBpile . EXPAND  1 3 

n«w_ll_oa»pile . SYNCHRONIZE 
n«w_ll_coaplla . TEST SYNCH 
naw_ll_cc*pila .  PARSE  18 

nov_ll_compile . LLMAIN  1 
new_ll_cc»pile . LL_COMPIL£ 
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Figure  25-9.  TCAT-PATH  Segment  Count  for  Each  Module  In  LL_COMPILE 
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Figure  25-10.  TCAT-PATH  Digraph  of  Function  LLFIND 


25-20 


PART  II  TCAT  Series  &  TDGen 


apg  (version  3.3  —  09/02/93]  —  paths  for  •nev_ll_complle . LLFIND" 


12  4  6 

123<{237]>8 
123  ({  237  )]  45 
123  ((237)]  46 
127  <{237  )>  8 
127  {{237}]  45 
127  {{  237  }]  46 
1  8 


Total  of  9  paths  for  ' nev_ll_compile . LLFIND' . 

Figure  25-11.  TCAT-PATH  All  Paths  for  LLFIND 


apg  (version  3.3  —  09/02/92]  —  paths  for  "nev_ll_eompile. LLFIND* 

12  4  5 
12  4  6 
1  8 


Total  of  3  paths  for  '  new_ll_co»pile .  LLFIND ' . 

Figure  25-12.  TCAT-PATH  Basis  Paths  for  LLFIND 


apg  [version  3.3  —  09/02/92]  —  paths  for  *  nev_ll_complle .  LLFIND  * 
Path  Analysis  Statistics 

File  nane i  new_ll_canpile . LLFIND . dig 


Number  of  nodes t  5 

Number  of  edges i  8 

Cyelomatie  number  (E  -  N  +  2): 

Number  of  paths:  9 

Average  path  length  (segments): 
Minimum  length  path  (segments): 
Maximum  length  path  (segments): 

Most  iteration  groups:  1 

Path  count  by  Iteration  groups: 

0  iteration  group(s):  3 

1  iteration  group(s):  6 

Stopped  at  1  iteration  groups 


5 


5.33 

3 

(Path 

7 

(Path 

(Path 

8) 

Figure  25-13.  TCAT-PATH  Path  Statistics  tor  LLFIND 
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pathcover  —  Path  Coverage  Utility.  [Release  1.2  —  9/91] 
(c)  Copyright  1991  by  Software  Research,  Inc. 


pathcover:  FIRST  INSTANCE  FOUND  BY  SEGMENT 


Module::  *new_ll_eonpila. LLFIND* 

Option::  *“f* 

« 

Path* 

Path 
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1 

12  1 
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2  3  2 
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3  4  4 
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2  5  8 

d 

OR 

pathcover : 

POPULATION  STATISTICS 

BY  SEGMENT 

Module::  *new_ll_caBpile. LLFIND* 

Option::  *-e* 

Segaent 

•  of  paths 

1 
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4 
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8 

1 

pathcover 
Nodule: : 

:  FIRST  INSTANCE  FOUND  BY  SEGMENT 
" new_ll_caaplle . LLFIND*  Option::  *-f* 

* 

Path*  Path 

1 

1 

12  4  5 

2 

2 

12  4  6 

3 

3 

1  8 

pathcover 

LAST  INSTANCE  FOUND  BY  SEGMENT 

Module: : 

"  new_ll_eoapile . LLFIND  *  Option:;  *-l* 

* 

Path#  Path 

1  1  12  4  5 

2  2  1  2  4  6 

3  3  18 


Figure  25-14.  TCAT-PATH  Path  and  Segment  Information  for  LLFIND 
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Ct  Test  Coverage  Analyser  Version  2.1  (9/91) 

(c)  Copyright  1991  by  Software  Research,  Inc. 

Module  *nev_ll_co®pile.BUILDRIGHT" :  26  paths,  8  were  hit  in  64  invocations. 
30.77*  Ct  coverage 

HIT/NOT-HIT  REPORT 
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Figure  25-15.  TCAT-PATH  Coverage  Report  for  BUILDRIGHT  using  testl.lex 
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Figure  25-16.  S-TCAT/Ada  Call  Graph  for  LL.TOKENS 
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Coverage  Analyzer.  [Release  8.2  for  SUN/UNIX  09/16/92] 
(c)  Copyright  1990  by  Software  Research,  Inc. 

Selected  SCOVER  System  Option  Settings: 


[-C] 

Cumulative  Report 

—  NO 

l-p] 

Past  History  Report 

—  NO 

[-n] 

Not  Hit  Report 

—  YES 

[-H] 

Hit  Report 

—  YES 

[-ah] 

Newly  Hit  Report 

—  NO 

[-am] 

Newly  Missed  Report 

—  NO 

t-h] 

Histogram  Report 

—  NO 

t-11 

Log  Seale  Histogram 

—  NO 

[-81 

Reference  Listing  SI 

—  NO 

Options  read:  2 

S-TCAT/C:  Coverage  Analyzer.  [Release  8.2  for  SUN/UNIX  09/16/92] 
(c)  Copyright  1990  by  Software  Research,  Inc. 

SI  Call-pair  Hit  Report . 


No.  Module  Name:  Call-pair  Coverage  Status: 


1  new_ll_compile . LL_CQMPILE 

2  new~ll~compile.UMAIN 

3  new_ll_coapile . READGRAM 

4  new_ll__compile .  BUILDRXGHI 

5  new~ll~compile .  BUILDSELECT 

6  new_ll_compile . PARSE 

7  nev_ll_compile . LLFIND 

8  n«w~ll“compile . LLNEXTTOKEN 

9  new_ll_t okens . ADVANCE 

10  new  11  tokens. SCAN  PATTERN 


11  new_ll_compile.GET_CHARACTER 

12  naw_ll~tokens . CHAR~ADVANCE 

13  nsw_ll~tokens . CURRENT_SYMBOL 

14  new~ll~compile .MAKE_T0KEN 

40  ll_sup_body . EMIT_SCAN_PROC 

41  ll~sup~body .  BUT  JSELECT 

42  ll_sup_body . EMIT_CHAR 

43  ll_8up_body . EMIT_PATTERN_MATCH 

44  ll_aup_body . EMIT_CONCAT_RI GHT 

45  ll_*up_body .  B4IT_C0NCAT_CASES 

Number  of  Call-pairs  Hit: 

Total  Number  of  Call-pairs: 

SI  Coverage  Value: 


All 

Call-pairs  Hit. 

SI 

-  1004 

All 

Call-pairs  Hit. 

SI 

-  1004 

All 

Call-pairs  Hit. 

SI 

-  1004 

All 

Call-pairs  Hit. 

SI 

-  1004 

All 

Call-pairs  Hit. 

SI 

-  1004 
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2  3  4 
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9  10 

All 
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SI 

-  1004 
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Figure  25-17.  S-TCAT/Ada  Call-Pair  Coverage  using  testl.iex 
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S-TCAT/C;  Coverage  Analyzer.  [Release  8.2  for  SUN /UN IX  09/16/92] 
(c)  Copyright  1990  by  Software  Research,  Xnc. 

SI  Not  Hit  Report. 


No.  Module  Name -.  Call-pair  Coverage  Status: 


1 

new_ll_coapile. LL_COMPILE 

All 

Call-pairs  Hit. 

SI 

- 

100% 

2 

new_ll_coapile . LIMAXN 

All 

Call-pairs  Hit. 

SI 

- 

1001 

3 

new_ll_coapile . REA0GRAM 

All 

Call-pairs  Hit. 

SI 

- 

100% 

4 

new_ll_compile . BUILDRIGHT 

All 

Call-pairs  Hit. 

SI 

- 

100% 

5 

new_ll_coapile . BUILDSELECT 

All 

Call-pairs  Hit. 

SI 

- 

100% 

6 

new_ll_compile . PARSE 

5 

6  7  11 

7 

new_ll_compile . LLFIND 

All 

Call-pairs  Hit. 

SI 

- 

100% 

8 

new~ll~compile . LLNEXTTOKEN 

All' 

Call-pairs  Hit. 

SI 

as 

100% 

9 

new_ll_tokens . ADVANCE 

5 

6 

10 

nev_ll_tokens . SCANJPATTERN 

1 

2  3  4 

5 

1 

6  10 

14 

15  16  25 

26 

27  29 

36 

39  40  44 

11 

new_ll_compila . GET_CHARACTER 

All 

Call-pairs  Hit. 

SI 

- 

100% 

12 

new~ll_tokens . CHAR_ADVANCE 

All  Call-pairs  Hit. 

SI 

- 

100% 

13 

new_ll_tokens . CURRENT_SYMBOL 

All 

Call-pairs  Hit. 

SI 

- 

100% 

14 

new_ll_cosipile .  MAKEJTOKEN 

7 

40 

ll_sup_body . EMIT_SCAN_PROC 

All 

Call-pairs  Hit. 

SI 

- 

100% 

41 

ll_aup_body . EMIT_SELECT 

4 

42 

ll_sup_body .  2XIT_CHAR 

All 

Call-pairs  Hit. 

SI 

m 

100% 

43 

ll_sup_body . EMIT_PATTERN_MATCH 

1 

2  3  4 

10 

11  12 

17 

18  20 

44 

ll_SUp_body .  IMIT_CONCAT_RIGHX 

1 

45 

ll_Sup_body . EMIT_CONCAT_CASES 

3 

5  6  7 

Number  of  Call-pairs  Not  Hit:  74 

Total  Number  of  Call-pairs:  162 

SI  Coverage  Value:  54.32% 


Figure  24-17  continued:  S-TCAT/Ada  Call-Pair  Coverage  Using  testl.lex 
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TCAT  Series  &  TDGen 


Coverage  Analyzer.  [Release  8.2  for  SUN/UNIX  09/16/92] 
(c)  Copyright  1990  by  Software  Research,  Znc. 

Selected  SCOVER  Sy steal  Option  Settings: 


[-c]  Cumulative  Report  —  NO 
[-p]  Past  History  Report  —  NO 
I-n]  Not  Hit  Report  —  YES 

[-H]  Hit  Report  —  YES 

[-nh]  Newly  Hit  Report  —  NO 
[-na]  Newly  Missed  Report  —  NO 
[-h]  Histogram  Report  —  NO 
[-1]  Log  Seale  Histogram  —  NO 
[-£]  Reference  Listing  SI  —  NO 
Options  read :  2 


S-TCAT/C:  Coverage  Analyzer.  [Release  8.2  for  SUN/UNIX  09/16/92] 
(o)  Copyright  1990  by  Software  Research,  Inc. 

SI  Call-pair  Hit  Report. 

No.  Module  Name:  Call-pair  Coverage  status: 


1 

nev_ll_compile . LLFXND 

All  Call-pairs  Hit. 

SI 

-  100% 

2 

new_ll_eompile . LLPRT STRING 

All  Call-pairs  Hit. 

SI 

-  100% 

3 

new_ll_oompile . LLPRTTOKKN 

4 

new_ll_eompile . LLSKIPTOKEN 

5 

new_ll_ea*pile . LLSXIPNODE 

6 

new_ll_eompile . LLSKIPBOTH 

7 

new_ll_eompile . LLFATAL 

8 

newJLl_eo*pile . GET_CHARACTKR 

All  Call-pairs  Hit. 

Si 

-  100% 

9 

new_ll_compile . CVT_STRING 

All  Call-pairs  Hit. 

Si 

-  100% 

10 

new_ll_co«pile . MAKE_TOXEN 

12  3  4 

5 

6 

11 

new_ll_oomp ile . LLNEXTTOKEN 

All  Call-pairs  Hit. 

Si 

-  100% 

12 

new_ll_oompile . BUXLDRXGHT 

All  Call-pairs  Hit. 

SI 

-  100% 

50 

ll_«up_body . EMZT_PATTERN_HATCH 

5  6  7  8 

9 

15  16 

22  23  24  25 

51 

ll_sup_body . EMIT_CHAR 

All  Call-pairs  Hit. 

SI 

-  100% 

52 

ll_Sup_body . EMIT_SELECT 

12  3 

53 

ll_sup_body .  H«tIscAN_PR0C 

All  Call-pairs  Hit. 

SI 

-  100% 

54 

ll_sup_body . XMXT_T0K£N 

All  Call-pairs  Hit. 

SI 

-  100% 

55 

ll_Sup_body .  INCLUDE_PATTERN 

1  3  4 

56 

ll_SUp_body . LO0K_AHEAD 

All  Call-pairs  Hit. 

SI 

-  100% 

57 

ll_sup_body . LOOK_OP_PATTERN 

All  Call-pairs  Hit. 

SI 

-  100% 

58 

ll_sup_body .  OPTION 

All  Call-pairs  Hit. 

SI 

-  100% 

59 

ll_sup_body . REPEAT 

All  Call-pairs  Hit. 

SI 

-  100% 

60 

ll_SUp_body . STORE_PATTERN 

61 

ll_actions .  LLTAXEACTX0N 

All  Call-pairs  Hit. 

SI 

-  100% 

Number  of  Call-pairs  Hit:  88 

Total  Number  of  Call-pairs:  253 

SI  Coverage  Value:  34.78% 

Figure  25-18,  S-TCAT/Ada  Call-Pair  Coverage  using  testl.lex  Accounting  for  All  Call-Pairs 
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S-TCAT/C :  Coverage  Analyzer.  [Release  8.2  for  SUN/UNIX  09/16/92] 
(c)  Copyright  1990  by  Software  Research,  Inc. 

SI  Not  Hit  Report. 


No. 

Module  Name:  Call-pair  Coverage  Status: 

1 

nsv_ll_ccapile .  LIu'IND 

All  Call-pairs  Hit.. 

SI  -  100% 

2 

nev_ll_campile . LLPRT STRING 

All  Call-pairs  Hit. 

SI  -  100% 

3 

nev_ll_ca»pile . LLPRTTOKEN 

1 

4 

nev_U_compile . LLSKIPTOKEN 

1  2 

5 

nev_ll_conpile . LLSXIPNODE 

1  2 

6 

nev_ll_coapile . LLSKZPBOTH 

12  3 

7 

nev_ll_oompile . LLFATAL 

1 

8 

n  ev~l l_camp lie . GET_CHARACTER 

All  Call-pairs  Hit. 

SI  -  100% 

9 

nev_ll_campile . CVT_STRING 

All  Call-pairs  Hit. 

SI  -  100% 

10 

nev_ll_compile . MAKE_TOKEN 

7 

11 

new_ll_oompile . LLNEXTTOKSN 

All  Call-pairs  Hit. 

SI  -  100% 

12 

new_ll_coapile . BUILDRIGHT 

All  Call-pairs  Hit. 

SI  -  100% 

13 

n«v_ll_coupile . BUILDSELECT 

All  Call-pairs  Hit. 

SI  -  100% 

14 

nsnr_ll_ocmpile .  READGRAM 

All  Call-pairs  Hit. 

SI  ->  100% 

48 

llsup_body . EMIT_CONCAT_CASES 

3  5  6  7 

49 

ll_sup_body .  BCT_CONCAT_RIGHT 

1 

50 

ll_SUp_body . EMIT~PATTERN_MATCR 

1  2  3  4; 

10  11  12 

17  18  ~  20 

51 

ll_sup_body .  BUT_CHAR 

All  Call-pairs  Bit. 

SI  -  100% 

52 

ll_sup_body . EMIT_SELECT 

4 

S3 

ll_SUp_body .  DaT_SCAN_PROC 

All  Call-pairs  Hit. 

SI  -  100% 

54 

ll_sup_body . EH1T_T0XXN 

All  Call-pairs  Hit. 

SI  -  100% 

55 

ll_SUp_body . INCLUDE_PATTERN 

2 

56 

ll_sup_body . LOOKAHEAD 

All  Call-pairs  Hit. 

SI  -  100% 

57 

ll_SUp_body . LOOK_UP_PATTERN 

All  Call-pairs  Bit. 

SI  -  100% 

58 

ll_sup_body . OPTION 

All  Call-pairs  Bit. 

SI  -  100% 

59 

ll_SUp_body . REPEAT 

All  Call-pairs  Hit. 

SI  -  100% 

60 

ll_SUp_body . ST0RE_P ATTERN 

1 

61 

ll_actions . LLT AKEACT I ON 

All  Call-pairs  Hit. 

SI  «  100% 

Number  of  Call-pairs  Not  Hiti 

165 

Total 

Number  of  Call-pairs: 

253 

SI  Coverage  Value: 

34.78% 

Figure  25-18  continued:  S-TCAT/Ada  Call-Pair  Coverage  using  test1.lex  Accounting  for  All 

Call-Pairs 
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Coverage  Analyzer.  [Release  8.2  for  SUN/UNIX  09/16/92] 
(e)  Copyright  1990  by  Software  Research,  Inc. 

Selected  SC OVER  System  Option  Settings: 


[-C] 

Cumulative  Report 

—  YES 

l-p] 

Past  History  Report 

—  NO 

[-U] 

Not  Hit  Report 

—  YES 

[-H] 

Hit  Report 

—  YES 

[-nh] 

Newly  Hit  Report 

—  YES 

[-nm] 

Newly  Missed  Report 

—  YES 

C-h] 

Histogram  Report 

—  NO 

(-11 

Log  Scale  Histogram 

--  YES 

t-xi 

Reference  Listing  SI 

—  NO 

Options  read:  6 

S-TCAT/C:  Coverage  Analyzer.  (Release  8.2  for  SUN/UNIX  09/16/92] 

(c)  Copyright  1990  by  Software  Research,  Inc. 

- - - - - - + 

I  |  Current  Test  I  Cumulative  Summary  I 


1  +- 

1  ! 

|  Module  Number  Of  | 

j  Name;  Call-pairs:  j 

No.  Of 
Invokes 

- - 

No.  Of  | 

Call-pairs  Sl%  | 
Hit  Cover  | 

No.  Of 
Invokes 

No.  Of  I 

Call-pairs  Sl%  j 
Hit  Cover  | 

+ - * - - - - - - 

' 

1  new  11  compile. LL_C0M 

1  | 

1 

1 

100.00  | 

2 

1 

100.00  | 

j  new  ll_campile . LLMAIN 

2  1 

1 

2 

100.00  j 

2 

2 

100.00  | 

|  new  11  compile. READGR 

2  I 

1 

2 

100.00  j 

2 

2 

100.00  j 

j  new  ll_eompile.BUILDR 

o  1 

64 

0 

100.00  j 

128 

0 

100.00  1 

j  new  11  compile . BUILDS 

0  I 

64 

0 

100.00  1 

128 

0 

100.00  j 

1  new  11  compile. PARSE 

11  1 

1 

7 

63.64  j 

2 

7 

63.64  j 

|  new_ll_oampile .  LLFIND 

0  1 

312 

0 

100.00  1 

510 

0 

100.00  1 

I  new  11  compile. LLNEXT 

0  I 

221 

0 

100.00  j 

355 

0 

100.00  j 

j  new  ll_tokens. ADVANCE 

6  I 

221 

5 

83.33  j 

355 

5 

83.33  j 

1  new  11  tokens. SCAN  PA 

44  j 

455 

22 

50.00  | 

712 

25 

56.82  | 

|  new_ll_oompile . GETCH 

0  1 

1385 

0 

100.00  j 

2250 

0 

100.00  j 

I  new  11_  tokens .  CHAR  AD 

o  1 

1238 

0 

100.00  j 

2018 

0 

100.00  j 

|  new  11  tokens . CURRENT 

o  1 

220 

0 

100.00  | 

353 

0 

100.00  | 

I  new  11  compile. MAKE_T 

7  | 

220 

7 

100.00  | 

353 

7 

100.00  j 

|  new_ll_compile . CVT_ST 

o  1 

220 

0 

100.00  j 

353 

0 

100.00  j 

|  new_ll_compile . EXPAND 

3  1 

461 

1 

33.33  j 

715 

1 

33.33  j 

|  nev_ll_canpile . MATCH 

0  | 

461 

0 

100.00  j 

715 

0 

100.00  j 

j  new_ll_compile . ERASE 

0  1 

707 

0 

100.00  j 

1105 

0 

100.00  j 

1  new  ll_tokens.L00X_AB 

0  I 

84 

0 

100.00  | 

123 

0 

100.00  j 

j  ll_actions . LLTAKEACTI 

.0  1 

429 

0 

100.00  1 

659 

0 

100.00  j 

|  ll_sup_body . TAIL 

18  | 

2 

0 

0.00  1 

2 

0 

0.00  | 

j  ll_sup_body . EMIT_ALT_ 

8  1 

12 

8 

100.00  j 

12 

8 

100.00  I 

* 

|  Totals 

238  | 

7569 

132 

55.46  | 

11884 

136 

57.14  | 

Figure  25-19.  S-TCAT/Ada  Call-Pair  Coverage  using  testl.lex  &  sample.lex 
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S-TCAT/C:  Coverage  Analyzer.  [Release  8.2  for  SUN /UNIX  09/16/92] 
(e)  Copyright  1990  by  Software  Research,  Inc. 


SI  Call-pair  Hit  Report. 


No.  Module  Name:  Call-pair  Coverage  Status: 


1 

new_ll_eompile . L1_C0MPILE 

All 

Call-pairs 

Hi..  SI 

-  100* 

2 

new~ll~compile . LLNAIN 

All 

Call-pairs 

Hit.  SI 

-  100* 

3 

new~ll_compile . READGRAM 

All 

Call-pairs 

Hit.  SI 

-  100* 

4 

new~ll_compile . BUILDRIGHT 

All 

Call-pairs 

Hit.  SI 

-  100* 

5 

new_ll_compile . BUILDSELECT 

All 

Call-pairs 

Hit.  SI 

-  100* 

6 

new_ll_coapile . PARSE 

1 

2  3 

4  8 

9  10 

7 

new_ll_coapile . LLFIND 

All. 

Call-pairs 

Hit.  Si 

-  100* 

8 

new_ll_compile . LLNEXTTOKEN 

All 

Call-pairs 

Hit.  SI 

-  100* 

9 

new_ll_tokens . ADVANCE 

1 

2  3 

4  5 

10 

new_ll_tokens . SCAN_PATTERN 

1 

2  3 

7  8 

9  11 

17 

19 

20  21 

22  23 

24  28 

31 

33 

34  37 

38  41 

42  43 

11 

new_ll_compile . GET_CHARACTER 

All 

Call-pairs 

Hit.  SI 

-  100* 

48 

ll_sup_body . RESOL VE_AMBIGUITY 

1 

2  9 

10  11 

12  13 

27 

29 

34  35 

36 

49 

ll_sup_body . RESTRICT 

1 

2  3 

4  5 

6  7 

50 

ll_sup_body . TAIL 

51 

ll_sup_body . EMIT_ALT_CASES 

All  Call-pairs 

Hit.  SI 

-  100* 

Number  of  Call-pairs  Hit: 
Total  Number  of  Call-pairs: 
SI  Coverage  Value: 


136 

238 

57.14* 


S-TCAT/C:  Coverage  Analyzer.  [Release  8.2  for  SUN/UNIX  09/16/92] 

(c)  Copyright  1990  by  Software  Research,  Inc. 

SI  Call-pair  Newly  Hit  Report.  # 


No. 

Module  Name:  Call-pair  Coverage 

:  Status: 

9 

new_ll_tokens . ADVANCE 

5 

10 

naw_ll_tokens . SCAN_PATTERN 

1 

2 

3 

48 

ll_sup_body . RESOLVE_AMB IGUITY 

1 

2 

9 

10 

11 

12 

13  27 

29 

34 

35 

36 

49 

ll_sup_body . RESTRICT 

1 

2 

3 

4 

5 

6 

7 

51 

ll”sup~body . EMIT_ALT^CASES 

1 

2 

3 

4 

5 

6 

7  8 

Figure  25-19  continued:  S-TCAT/Ada  Call-Pair  Coverage  using  testl.lex  &  sample.lex 
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S-TCAT/C:  Coverage  Analyzer.  [Release  8.2  for  SUN/UNIX  09/16/92] 
(e)  Copyright  1990  by  Software  Research,  Inc. 


SI  Not  Hit  Report. 

No.  Module  Name:  Call 

1  new_ll_coapile . LL_COMPILE 

2  nev_ll_eompile . LLMAIN 

3  new_ll_compile . READGRAM 

4  new_ll_compile . BUILDRIGHT 

5  new  ll_conpile . BUILDSELECT 

6  new_ll_compile . PARSE 

7  new_ll_compile . LLFIND 

8  new~ll”compile.LLNEXTTOKEN 

9  new_ll~tokens. ADVANCE 

10  nev_ll_tokens . SC AN_PATTERN 


new_ll_compile . GET_CHARACTER 


■pair  Coverage  Status: 
All  Call-pairs  Hit. 
All  Call-pairs  Hit. 
All  Call-pairs  Hit. 
All  Call-pairs  Hit. 
All  Call-pairs  Hit. 

5  6  7  11 

All  Call-pairs  Hit. 
All  Call-pairs  Hit. 

6 

4  5  6  10 

25  26  27  29 

44 

All  Call-pairs  Hit. 


SI  -  1008 


ll_sup _body . EMIT_SCAN_PROC 
ll_sup_body . EMIT_SELECT 
ll_Sup_body . EMIT_CHAR 
ll~sup_body . EMIT_PATTERN_MATCH 
ll~sup_body . EMIT_CONCAT_RIGHT 
ll~sup_body .  DilT _CONCAT_CASES 
ll~sup  J>ody . CVT_STRING 
ll_sup_body. CVT_ASCII 
ll_aup_body . RESOLVE_AMBIGUITY 


4  9  ll_Sup_body . RESTRICT 

5  0  ll_sup_body . TAIL 

51  ll_sup_body .  IMIT_ALT_CASES 

Number  of  Call-pairs  Not  Hit: 
Total  Number  of  Call-pairs: 

SI  Coverage  Value: 


All  Call- 

4 

All  Call-pairs  Hit.  SI  ”  100% 

2  11  12  13  14  17  18 

All  Call-pairs  Hit.  SI  -  100% 

3  5 

All  call-pairs  Hit.  SI  -  100% 

All  call-pairs  Hit.  SI  -  100% 

3  4  5  6  7  B  14  15  16 

17  18  19  20  21  22  23  24  2 

26  30  31  32  33 

8  9  10  11  12  13 

123456789 

10  11  12  13  14  15  16  17  1 

All  Call-pairs  Hit.  SI  -  100% 

102 

238 

57.14% 


-pairs  Hit.  SI  “  100% 


S-TCAT/C:  Coverage  Analyzer.  [Release  8.2  for  SUN/UNIX  09/16/92] 
(e)  Copyright  1990  by  Software  Research,  Inc. 

SI  Call-pair  Newly  Missed  Report. 

No,  Module  Name:  Call-pair  Coverage  Status: 

10  new_ll_tokens.SCAN_PATTERN  19  20  21 

40  ll_sup~body .  H!IT_SCAN_PROC  3 


Figure  25-19  continued:  S-TCAT/Ada  Call-Pair  Coverage  using  testl.lex  &  sample.lex 
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S-TCAT/C:  Coverage  Analyzer.  (Release  8.2  for  SUN/UNIX  09/16/92] 

(e)  Copyright  1990  by  Software  Research,  Inc. 

Call-pair  Level  Histogram  for  Module:  new_ll_compile.LL_COMPILE 

+ - + 

|  Logarithm  of  Executions,  Normalized  to  Maximum 
j  (Maximum  -  2  Hits) 

Call-pair  Number  Of  | 

Number  Executions  > - 1 - —10 — — — 20 - 30 - 40 — 80-100 


Average  Hits  per  Executed  Call-pair:  2.0000 

SI  Value  for  this  Module:  100.0000 


S-TCAT/C:  Coverage  Analyzer.  (Release  8.2  for  SUN/UNIX  09/16/92] 
(c)  Copyright  1990  by  Software  Research,  Inc. 

Call-pair  Level  Histogram  for  Module:  new_ll_eoapile . BUILDSELECT 
No  call-pairs  present  or  hit 

S-TCAT/C:  Coverage  Analyzer.  [Release  8.2  for  SUN/UN LX  09/16/92] 
(c)  Copyright  1990  by  Software  Research,  Inc. 

Call-pair  Level  Histogram  for  Module:  new_ll__campile . PARSE 


1  Logarithm  of  Executions,  Normalized  to  Maximum 
j  (Maximum  “  1105  Bits) 

Call-pair  Number  Of  j 

Number  Executions  > - 1 - 10 - 20 - 30 - 40 — 80-100 


(*  -  Zero  Hits) 

Average  Hits  per  Executed  Call-pair:  405.4286 
SI  Value  for  this  Module:  63.6364 


Figure  25-19  continued:  S-TCAT/Ada  Call-Pair  Coverage  using  testUex  &  sample.lex 
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Figure  25-20.  TSCOPE  Dynamic  Display  of  Coverage  on  Directed  Graph  for  LLFIND 


Figure  25-21.  TSCOPE  Dynamic  Display  of  Coverage  Accumulation  for  LLFIND 
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{%c  Values  file  1:  for  variable  Busbar  of  initial  TDGen  executions.  ) 

expx  (t  expr)(%  op)(%  expr )  (*  identifier)  (*  real_no) 

op  +  -  /  * 

identifier  variablel  variable!  (%  alpha  6) 

real.no  {«  real  4.6]  (t  integl)E+{%  integ!)  (t  integl]E-{%  integ!) 

integl  [%r  1..100) 

integ!  ( tr  3 . .  6 ) 


(tc  Values  file  2:  for  last  two  executions  of  TDGen.  ) 

expr  variablel  variable!  (t  alpha  6}  (%  real  4.6}  {%  Integl )E+[%  ii 

op  +  -  /  • 

identifier  variablel  variable!  (%  alpha  6) 

real.no  [t  real  4.6)  (t  integl )E+(t  integ!)  {%  integl)E-(*  integ!) 

integl  [*r  1..100) 

integ2  (tr  3.. 6} 


(tc  Template  file:  Produces  arithmetic  expression  of  varying  ] 
(*c  complexity  for  use  in  testing  a  generated  lexical  analyzer.  ) 

(%  expr  ) 


Figure  25-22.  TDGen  Sample  Value  and  Template  Files 


Field 

No.  Table 
Entries 

Cumulative  Total 

Combinations 

%  expr 

3 

3 

• 

t  op 

4 

12 

%  identifier 

3 

36 

*  real.no 

3 

108 

%  integl 

100 

10800 

%  integ2 

4 

43200 

Figure  25-23.  TDGen  Table  of  Sequential  Combinations  for  Initial  Flies 
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(%  real_no} 

{%  expr}{%  opJC*  expr) 

{%  identifier} 

{%  expr}{%  op}{%  expr} 

{%  identifier} 

{%  identifier} 

{%  expr}{%  op}{%  expr) 

{%  real_no} 

{%  real_no} 

{«  identifier} 

Figure  25-24.  TDGen  Output  of  First  Random  Execution 


3E+6 

{%  real  4. 6) -variablel 
RSBEZ4 

{%  integl)E-(%  integ2}-{%  integl}E-{%  integ2) 

variable2 

variables 

{%  identifier}* {%  real_no}/{*  identifier ]/{%  identifier) 

21E+4 

47E-6 

variablel 


Figure  25-25.  TDGen  Output  After  3  Executions  with  1st  Value  File 


3E+6 

3093 . 7  03258-variablel 

RSBEz4 

53E-4-83E-6 

variable2 

variables 

G36dk5*26E-5/claHEJ/variable2 

21E+4 

47E—6 

variablel 


Figure  25-26.  TDGen  Output  After  2  Executions  with  2nd  Value  File 
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26.  TST 

The  Ada  Test  Support  Tool  (TST)  is  government  owned.  Designed  to  facilitate  the  test¬ 
ing  of  Ada  subprograms  and  task  entry  points,  it  provides  test  driver  generation  with  test 
data  generation  for  program  unit  parameters.  TST  operates  in  batch  or  interactive  mode. 


26.1  Tool  Overview 

TST  was  developed  by  Intermetrics,  Inc.  under  contract  to  the  Software  Technology  for 
Adaptable,  Reliable  Systems  (STARS)  Foundations  program.  It  leverages  off  technology 
developed  for  the  Ada  Test  and  Analysis  Tools  (ATEST)  Intermetrics  previously  built  for 
the  Worldwide  Military  Command  and  Control  (WWMCCS)  Information  System  (W1S) 
program.  The  first  version  of  this  toolset  became  available  in  1989.  It  is  compiler  indepen¬ 
dent  and  designed  to  be  portable.  Intermetrics  has  hosted  TST  on  the  Alsys  PC/AT,  Alsys 
Sun,  and  DEC  Ada  compilers.  It  is  available  at  no  charge  from  the  STARS  Foundation  Ar¬ 
chive. 

The  evaluation  was  performed  on  TST  version  2  running  in  a  VAX/VMS  environment. 

TST  consists  of  three  parts:  a  Shell,  Source  Instrumentor,  and  Testing  Subsystem.  When 
used  interactively,  the  Shell  provides  a  test  environment  where  the  user  can  set  various  de¬ 
fault  parameters,  such  as  the  name  of  a  separate  directory  to  hold  all  the  files  generated  dur¬ 
ing  instrumentation.  It  allows  the  user  to  invoke  the  Source  Instrumentor  and  handle  the 
compilation,  linking,  and  execution  of  the  instrumented  code.  The  Shell  also  provides  for 
the  management  of  internal  TST  files,  and  screen  and  terminal  handling. 

Testing  starts  by  invoking  the  Source  Instrumentor  to  insert  statements  that  allow  call¬ 
ing  contained  subprograms  and  task  entry  points  into  a  library  unit  under  test  The  instru¬ 
mentation  caters  for  reading  and  writing  of  parameter  values,  assertion  testing,  and  logging 
of  test  results  and  program  execution  information.  Both  the  specification  and  body  of  the 
unit(s)  under  test  are  submitted  to  the  Source  Instrumentor.  (Although  statements  are  not 
inserted  in  package  specifications,  the  instrumentor  does  extract  some  information  from 
them.)  Units  containing  type  declarations  that  are  used  by  the  unit  under  test  are  also  need¬ 
ed.  The  user  is  required  to  submit  additional  information  for  testing  generic  items.  For  ex¬ 
ample,  instrumentation  of  a  generic  package  requires  the  user  to  provide  actual  type  and 
subprogram  names,  whereas  a  generic  formal  type  or  subprogram  requires  a  package  name 
and  then  the  name  of  the  actual  type  or  subprogram.  In  the  case  of  generic  declarations,  the 
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user  must  provide  type  and  subprogram  names  for  generic  parameters.  One  instantiation  of 
each  generic  unit  is  generated  using  specified  names. 

Statements  for  automatic  test  data  generation  for  predefined  parameter  types  are  auto¬ 
matically  included  in  the  source  code.  The  user  is  queried  whether  test  data  generation  for 
user-defined  types  should  be  included.  Test  data  generation  is  performed  in  two  ways.  In 
one  case,  TST  generates  all  possible  values  for  a  parameter  (or  the  first  and  last  values  for 
floating  point  types).  In  the  other,  the  user  specifies  that  all  possible  values  are  divided  into 
a  given  number  of  partitions,  and  TST  then  selects  the  first,  middle,  and  last  value  from 
each  partition.  The  user  is  responsible  for  ensuring  that  the  number  of  values  generated  is 
not  sufficient  to  cause  the  Ada  exception  Storage  Error  to  be  raised.  This  automatic  test 
data  generation  is  not  available  for  task,  private,  or  limited  private  types.  When  requesting 
generation  for  unconstrained  types,  such  as  an  unconstrained  array,  record,  or  string,  the 
user  must  give  a  constraint  Optionally,  constraints  may  also  be  given  for  character  types. 


Finally,  the  Source  Instrumentor  generates  a  test  driver  to  call  the  routines  contained  in 
the  library  unit.  This  test  driver  is  included  at  the  end  of  the  instrumented  source  file.  At  the 
user’s  option,  the  Source  Instrumentor  also  prepares  a  pretty  printed  source  code  listing. 
This  listing  includes  breakpoint  numbers  that  are  used  in  path  analysis  reports  to  identify 
the  statements  that  were  executed. 


Once  the  generated  testbed  has  been  compiled  and  linked,  it  can  be  invoked  under  TST 
and  then  TST  hands  control  to  the  Testing  Subsystem.  This  subsystem  provides  a  dual-win¬ 
dow  user  interface.  The  user  interacts  with  TST  through  the  Dialogue  Window,  while  the 
Display  Window  provides  useful  information  in  the  form  of  declarations  for  all  the  routines  # 

that  may  be  tested  and  for  current  assertions.  (A  TST  option  provides  for  handling  data  out¬ 
put  to  the  screen  from  the  unit  under  test  This  option  allows,  for  example,  directing  the  unit 
output  to  the  Testing  Subsystem  windows,  or  to  another  window  superimposed  over  these.) 

The  user  is  asked  for  a  test  identification,  and  the  name  of  the  Test  Data  File  (TDF)  that  ® 

contains  the  assertions  and  calls  that  will  be  used  to  test  the  unit  If  the  named  TDF  does 
not  exist,  the  Testing  Subsystem  saves  the  user’s  subsequent  test  input  in  the  named  file  so 
that  a  test  run  can  be  easily  repeated.  TDFs  can  also  be  created  or  modified  outside  of  TST. 

Testing  proceeds  by  calling  procedures,  functions,  and  entry  points  within  the  unit  under  ^ 

test  and  making  assertions  about  the  output  Each  routine  is  identified  using  the  numbers 
given  in  the  display  window  and,  when  the  user  requests  its  call,  TST  queries  for  input  pa¬ 
rameter  values.  He  may  enter  actual  parameter  values  using  named  or  positional  notation. 

Alternatively,  the  user  may  request  automatic  test  data  generation  for  a  parameter.  The  user  ^ 
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must  also  specify  values  for  OUT  mode  parameters.  These  will  be  used  for  constraints 
where  required,  for  example,  for  string  OUT  parameters.  The  Testing  Subsystem  calls  the 
associated  routine  for  each  generated  combination  of  test  data.  Once  a  test  is  complete,  the 
values  of  OUT  and  IN  OUT  parameters,  or  function  results,  are  displayed  in  the  Dialogue 
Window. 

Assertions  can  be  given  to  check  these  test  results.  These  assertions  may  be  global,  that 
is,  valid  from  the  time  the  assertion  is  given  until  either  the  end  of  the  test  session  or  the 
deletion  of  that  assertion.  Alternatively,  local  assertions  are  valid  only  for  the  next  call  com¬ 
mand,  or  the  multiple  calls  of  a  single  routine  that  may  be  incurred  by  test  data  generation. 
The  validity  of  assertions  is  not  checked  when  defined,  but  only  when  an  assertion  is  eval¬ 
uated  for  the  specified  results.  When  an  assertion  fails,  a  message  is  printed  to  the  screen 
and  the  testbed  will  either  continue,  abort  and  start  report  generation,  or  query  the  user 
whether  to  continue  or  abort  depending  on  how  an  Assertion  Handling  flag  is  set. 

The  user  can  give  a  number  of  other  commands  to  the  Test  Subsystem.  These  are  used, 
for  example,  to  control  the  display  area,  manage  assertions,  and  control  the  handling  of  any 
screen  output  generated  by  the  routine  under  test. 

TST  automatically  generates  a  TST  report  at  the  end  of  a  test  execution.  As  well  as  gen¬ 
eral  identification  information,  this  report  lists  the  Ada  declarations  for  all  visible  proce¬ 
dures,  subprograms,  and  entry  points  of  the  unit  under  test,  test  data  that  was  generated,  and 
results  of  invoked  routines  and  associated  assertions.  The  user  may  request  a  path  analysis 
report  to  be  included.  This  report  provides  a  trace  of  the  execution  history  and  an  execution 
summary  report  that  lists  the  number  of  times  each  statement  (or  group  of  consecutive  state¬ 
ments)  was  executed. 


26.2  Observations 

Ease  of  use.  The  on-line  help  provides  summaries  of  Shell  and  Testing  Subsystem  com¬ 
mands  that  are  very  helpful.  A  simple  “?”  provides  a  list  of  currently  available  commands. 

Limited  tailorability  is  available.  The  help  file  format  is  tailorabie,  allowing  the  user  to 
modifying  existing  messages  or  add  new  messages.  The  user  can  define  the  type  of  terminal 
being  used  via  an  ANSI  X3.64  Compatible  Virtual  Terminal  Package  Terminal  capabilities 
files,  a  variation  of  the  TERMCAP  developed  in  the  Berkeley  extensions  to  Unix.  This  al¬ 
lows,  for  example,  user-defined  function  keys. 
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Documentation  and  user  support.  The  installation  instructions  received  with  the  software 
from  the  STARS  Foundation  Repository  had  some  minor  omissions.  The  TST  documentation  itself 
is  easy  to  follow  and  helpful.  No  direct  support  for  the  tool  is  available,  although  Intermetrics  an¬ 
swered  questions  that  arose  during  this  examination. 

Instrumentation  overhead.  The  instrumentation  performed  by  TST  imposes  a  substantial 
overhead.  The  degree  of  code  expansion  largely  depends  on  the  number  and  type  of  type  defini¬ 
tions  encountered,  and  number  of  subprogram  units  being  tested.  In  the  case  of  the  Ada  Lexical 
Analyzer  Generator,  the  source  code  size  of  a  library  routine  containing  the  function  LLFIND 
alone  was  four  blocks.  When  instrumented  for  full  test  data  generation,  this  size  increased  to  124 
blocks. 

However,  since  bottom-up  testing  requires  units  to  be  tested  independently  or  in  small  groups, 
with  careful  partitioning  of  the  code,  the  instrumentation  overhead  may  not  be  a  significant  prob¬ 
lem. 


Ada  restrictions.  The  generated  control  program  is  subject  to  the  same  restrictions  that  any 
program  would  be  in  calling  package  subprograms.  For  example,  values  cannot  be  given  to,  or  re¬ 
ceived  from,  objects  declared  as  private.  TST  imposes  additional  restrictions  largely  to  do  with  the 
format  and  content  of  input  data  to  the  subprograms  being  tested.  These  restriction  include  the  fol¬ 
lowing:  (1)  values  must  be  given  for  all  parameters  that  do  not  have  defaults  and  named  notation 
is  required  for  use  of  defaults;  (2)  all  parameter  values  and  assertion  values  must  be  literal  values, 
and  (3)  test  data  generation  is  not  supported  for  tasks  types,  private  types,  limited  private  types,  or 
records  types  with  nested  variant  parts. 

Problems  encountered.  A  couple  of  problems  during  instrumentation  required  minor  manual 
editing  before  the  instrumented  source  code  would  compile.  After  instrumentation,  some  Ada  use 
statements  had  to  be  manually  inserted  to  cater  for  inserted  with  statement  Some  problems  were 
experienced  generating  test  data  for  string  subtypes. 


26.3  Recent  Changes 

Intermetrics  has  continued  development  of  TST.  The  augmented  version  is,  however,  a  propri¬ 
etary  Intermetrics  tool. 
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26.4  Sample  Outputs 

Figures  26-1  through  26-6  provide  sample  outputs  from  TST. 
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Test  Support  Tool  -  Version  2.0  Page:  1 

Test  Configuration  Report 


Program  Under  Test: 
Test  Date: 

Test  Day: 

Test  Time: 

Data  File: 


LL_C0MPILE3 

09/03/92 

THURSDAY 


12:57:50 
RUNS . TDF 


Test  ID: 


Test  LLFIND  Sun  3 


Executable  File: 


Defaults : 


Naae  -  NOT  AVAILABLE 
Date  -  ???????? 

Time  -  ???????? 

TST_DIR  -  [ADATEST . TST . TSTDIR] 
BPT_CPL  -  60 
BPT_LPP  “  54 

ASSERTIONHANDLING  -  CONTINUE 
SCREEN  ECHO  -  ON 


Test  Support  Tool  -  Version  2.0 
Tast  Configuration  Report 


Page: 


2 


1  function  LLFIND ( 

ITEM  :  LLSTRXNGS; 

WHICH  :  LL STYLE)  return  INTEGER; 

2  procedure  LLMAIN; 


Figure  26-1.  TST  Test  Configuration  File  for  Function  LLFIND 
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Test  Support  Tool  -  Version  2.0  Page:  3 

Parameter  Report 

GLOBAL  Assertion  1)  1  <  33 
Unit  Under  Test:  (  2) 
procedure  LLMAIN; 


LOCAL  Assertion  1)  1  -  12 
Unit  Under  Test:  (1) 

function  LLFIND( 

ITEM  :  LL STRINGS ; 

WHICH  :  LLSTYLE)  return  INTEGER; 

Parameter  Entering  Value  Exiting  Value 


ITEM  'Identifier  ..." 

WHICH  GROUP 

< RETURN  VALUE >  12 


Unit  Under  Test:  (  1) 

function  LLPIND{ 

ITEM  i  LLSTRINGS ; 

WHICH  :  LLSTTLE)  return  INTEGER; 


Test  Data  Automatically  Generated 
WHICH  ->  * 

Parameter  Entering  Value  Exiting  Value 


ITEM 

WHICH 

<RETURN_VALUE> 

•ASCII 

LITERAL 

a 

10 

ITEM 

•ASCII 

■ 

WHICH 

NONTERMINAL 

<RETURN_VALUE> 

0 

ITEM 

■ASCII 

■ 

WHICH 

GROUP 

<RETURN_VALUE> 

0 

ITEM 

■ASCII 

B 

WHICH 

ACTION 

Figure  26-2.  TST  Parameter  Report  for  Function  LLFIND 
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< RET URN  VALUE >  0 


ITEM  'ASCII  . . . ■ 

WHICH  PATCH 

< RET URN  VALUE >  0 


LOCAL  Assertion  2)  1  -  0 
Unit  Under  Test:  (  1) 

function  LLFIND( 

ITEM  :  LLSTRINGS; 

WHICH  :  LLSTYLE)  return  INTEGER; 


Test  Data  Automatically  Generated 
WHICH  ->  * 


Parameter 

Entering  Value 

Exiting  Value 

ITEM 

WHICH 

<RETURN_VALUE> 

9 

LITERAL 

a 

0 

ITEM 

WHICH 

<RETUKN_VALUE> 

■ 

NONTERMINAL 

* 

0 

ITEM 

WHICH 

<RETURN_VALUE> 

■ 

GROUP 

n 

0 

ITEM 

WHICH 

<SETURN_VALUE> 

• 

ACTION 

9 

0 

ITEM 

WHICH 

<RETURN_VALUE> 

m 

PATCH 

• 

0 

LOCAL  Assertion 
Unit  Under  Test: 

3)  I  -  12 
(  1) 

function  LLFIND ( 

ITEM  :  LLSTRINGS; 

WHICH  >  LLSTTLE)  return  INTEGER; 

Parameter 

Entering  Value 

Exiting  Value 

ITEM 

WHICH 

<  RETURN_VALUE > 

■ASCII 

LITERAL 

9 

10 

***  LOCAL  Assertion  3)  1  -  12  Failed 

Figure  26-2  continued:  TST  Parameter  Report  for  Function  LLFIND 


PART  II 


Test  Support  Tool  -  Version  2.0  Page:  6 

Execution  History  Report 


Begin  LL  COMPILES . LLMAIN . READGRAM. BUILDRIGHT 

[77] 

Begin  LL_C0MPILE3 . LLMAIN . READGRAM . BUILDSELECT 

[52-55]  [55]  [55]  [55]  [55J  [55]  [55]  [55]  [55]  [55]  [55]  [55] 
[55]  [55]  [55]  [55]  [55]  [55]  [55]  [55-58]  [60]  [55] 

End  LL_CQMPILE3. LLMAIN. READGRAM. BUILDSELECT 
Resune~LL_COMP  ILE3  .  LLMAIN .  READGRAM .  BUILDRIGHT 

[78] 

End  LL_C0MPXLE3 . LLMAIN . READGRAM . BUILDRIGHT 
Begin  LL_COMPZLE3 . LLFXND 

[1-6]  [10]  [10]  [7-8] 

End  LL_C0MPILE3. LLPIND 
Begin  LL_COMPILE3 . LLFIND 
[1-6]  [10]  [7-8] 

End  LL_C0MPILE3 . LLFIND 
Begin  LL_COMPILE3 . LLFIND 
[1-6]  [10]  [7]  [9] 

End  LL_C0MPILE3. LLFIND 
Begin  LL_COMPILE3 . LLFIND 
[1-6]  [10]  [7]  [9] 

End  LL_CQMPILE3. LLFIND 
Begin  LL_COMPILE3 . LLFIND 
[1-6]  [101  [7]  [9] 

End  LL_C0MPILE3 .LLFIND 
Begin  LL  COMPXLE3 . LLFIND 
[1-6]  [10]  [7]  [9] 

End  LL_CQMPILE3 . LLFIND 
Begin  LL_CQHPILE3 . LLFIND 
[1-6]  [11] 

End  LL_C0MPILE3. LLFIND 
Begin  LL_COMPILE3 . LLFIND 
[1-6]  [11] 

End  LL_C0MPILE3 .  LLFIND 
Begin  LL  C0MPILE3 . LLFIND 
[1-6]  [11] 

End  LL_C0MPILE3. LLFIND 
Begin  LL_COMPILE3 . LLFIND 
[1-6]  [11] 

End  LL_C0MPILE3. LLFIND 
Begin  LL  COMPILES . LLFIND 
[1-6]  [11] 

End  LL_C0MPILE3. LLFIND 
Begin  LL  COMPILES . LLFIND 
[1-6]  [10]  [7-8] 

End  LL  C0MPILE3. LLFIND 


Figure  26-3.  TST  Execution  History  Report  for  Function  LLFIND 
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Test  Support  Tool  -  Version  2.0 
Execution  Sumoary  Report 


Statement 


LL_C0MP  ILE3 
[1-3] 
[4-51 
[«] 

[7] 

[8] 

[91 

[101 

[HI 

[121 

[13-151 

[16-191 

[20-23] 

[24] 

[25-28] 

[29-321 

[33] 

[34-35] 

[36] 

[37] 
[381 
[39] 
[401 

[41] 

[42] 
[43-44] 
[45-47] 
[48-49] 
[50-51] 
[52-53] 

[54] 

[55] 
[56-58] 

[59] 

[60] 
[61-54] 
[65-70] 
[71] 
[72-74] 
[75-78] 


Execution  count 


12 

65 

50 

7 

3 

4 

5 

5 
0 

64 

174 

50 

61 

46 

13 

4 

0 

174 

79 

95 

174 

144 

30 

174 

0 

64 

227 

64 

1 

32 

640 

32 

6 

26 

1 

64 

1 

26 

1 


Page i  21 


Figure  26-4.  TST  Execution  Summary  Report  for  Function  LLFIND 
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GLOBAL_ASSERT(  1  <  33  ) 

2 
1 

LOCAL_ASSERT(  1  -  12  > 

CALL_ROUTINE  1  ( 

ITEM  ->  ■Identifier 
WHICH  ->  GROUP 

); 

—  *  1 

CALL_ROUTINE  1  ( 
item” “ >  "ASCII 
WHICH  ->  LITERAL 

): 

CALL_ROUTINE  1  ( 

ITEM~->  "ASCII 
WHICH  ->  PATCH 

); 

— *  l 

LOCAL_ASSERT (  1  -  0  ) 

CALL_ROUTINE  1  ( 

ITEM  ->  " 

WHICH  ->  LITERAL 

)t 

LOCAL_ASSERT(  1  -  0  ) 

CALL_ROUTINE  1  ( 

ITEM  ->  * 

WHICH  ->  NONTERMINAL 

)/ 

LOCAL_ASSERT(  1  -  0  ) 

CALL_ROUTINE  1  ( 

ITEM  ->  " 

WHICH  ->  GROUP 

)  f 

LOCAL_ASSERT (  1  -  0  ) 

CALL_ROUTINE  1  ( 

ITEM  ->  " 

WHICH  ->  ACTION 
)  ! 

LOCAL_ASSERT(  1  -  0  ) 

CALL_ROUTINZ  1  ( 

ITEM~->  " 

WHICH  ->  PATCH 

); 

— •  1 

LOCAL_AS SERT (  1  -  12  ) 

CALL_ROUTINE  1  ( 

ITEM  ->  "ASCII 
WHICH  ->  LITERAL 

); 


Figure  26-5.  TST  Sample  Test  Data  File  for  Function  LLFIND 
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With  LL_DECLARAT IONS ,  INTEGER_TEXT_IO ,  TEXT_XO; 
package  LL_COMPILE3  is 

use  LL_DECLARAT IONS ,  INTEGER_TEXT_IO ,  TEXT_IO; 

PARSING_ERROR :  exception;  —  for  fatal  parsing  errors 
type  LLSTYLE  is  (LITERAL,  NONTERMINAL,  GROUP,  ACTION,  PATCH); 
type  LLSYMTAB ENTRY  is  —  for  symbol  table  entries 
record 

RET:  LLSTRINGS ;  —  literal  string  or  group  identifier 

KIND)  LLSTYLE;  —  literal  or  group 
end  record; 

LLSYMBOLTABLEi  array  (  1  . .  LLTABLESIZE  )  of  LLSYMTAB ENTRY; 

—  the  symbol  table  for  literal  terms 
function  LLFIND(  ITEM;  LLSTRINGS;  WHICH ■  LLSTYLE  )  return  INTEGER; 
procedure  LLMAIN; 
end  LL.C0MPILE3; 

With  LLJJECLARATIONS,  INTEGER_TEXT_IO,  TEXTJEO; 
package  body  LL_C0MPILE3  is 

use  LL_DECLARATIONS,  INTEGER_TEXT_IO ,  TEXT_IO; 

function  LLFIND(  ITEM!  LLSTRINGS;  WHICH i  LLSTYLE  )  return  INTEGER  is 

—  Find  item  in  symbol  table  —  return  index  or  0  if  not  found. 

—  Assumes  symbol  table  is  sorted  in  ascending  order. 

LOW,  MIDPOINT,  HIGH!  INTEGER; 

begin 

LOW  :«  1; 

HIGH  LLTABLESIZE  +  1; 
while  LOW  /-  HIGH  loop 

MIDPOINT  i-  (HIGH  +  LOW)  /  3; 
if  ITEM  <  LLSYMBOLTABLE (MIDPOINT) .KEY  then 
HIGH  !-  MIDPOINT ; 

elaif  ITEM  -  LLSYMBOLTABLE (MIDPOINT) .KEY  then 
if  LLSYMBOLTABLE (MIDPOINT) .KIND  -  WHICH  then 
return (  MIDPOINT  }; 

else 

return(  0  ); 
end  if; 

else  —  ITEM  >  LLSYMBOLTABLE  (MIDPOINT)  .KEY 
LOW  !-  MIDPOINT  +  1; 
end  if; 
end  loop; 

return (  0  );  —  item  ip  not  in  table 

end  LLFIND; 

procedure  LLMAIN  is 

end  LLMAIN; 
end  LL  COMPILES ; 


Figure  26-6.  TST  Function  LLFIND 
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27.  Test/Cycle  and  Metrics  Manager 

Test/Cycle  supports  the  definition  of  functional  requirements  and  validation  criteria 
such  as  test  plans,  test  runs,  and  test  cases.  It  provides  users  with  a  graphical  object-oriented 
framework  for  developing  test  plans,  managing  of  software  testing  efforts,  and  problem  re¬ 
porting.  The  concept  of  software  builds  supports  incremental  development. 

Metrics  Manager  is  a  measurement  system  that  focuses  on  productivity  and  quality  im¬ 
provement.  It  is  supported  with  an  industry  data  base  of  software  metrics  that  allows  Met¬ 
rics  Manager  users  to  assess  their  productivity  in  relation  to  other  organizations.  Currently 
with  data  from  over  two  hundred  projects  from  some  twenty  five  Fortune  500  organiza¬ 
tions,  this  database  is  expected  to  double  in  size  in  the  near  future. 


27.1  Tool  Overview 

Test/Cycle  and  Metrics  Manager  are  marketed  by  Computer  Power  Group,  Inc.  This  or¬ 
ganization  primarily  markets  testing  services,  consulting,  and  training  in  software  testing 
and  quality  assurance.  It  is  a  founding  member  of  the  Quality  Assurance  Institute  and  still 
serves  as  a  board  member.  Computer  Power  Group  continues  to  undertake  research  into 
software  quality  measurement  in  conjunction  with  the  Quality  Assurance  Institute  and  oth¬ 
er  groups. 

Test/Cycle  was  released  in  1990  and  is  used  at  some  10  to  15  sites.  Metrics  Manager 
first  became  available  in  1989  and  is  used  at  30  to  35  sites.  Both  tools  are  PC  based  and  run 
under  MS-DOS;  Test/Cycle  is  available  under  Microsoft  Windows.  Both  tools  are  avail¬ 
able  under  IBM’s  Ad/Cycle  tool  set.  Metrics  Manager  can  import  project  data  from  Project 
Workbench;  it  also  supports  a  bidirectional  interface  to  Project  Bridge  for  exchange  of 
function  point  data.  At  the  time  of  examination,  the  price  of  Test/Cycle  started  at  $3,500. 
Metrics  Manager  is  now  marketed  by  ABT  Technologies  and  its  price  currently  starts  at 
$14,950;  this  includes  a  consulting  project  to  set  up  appropriate  metrics  for  the  client.  The 
examinations  were  performed  on  Test/Cycle  version  3.02  and  Metrics  Manager  version 
2.02. 
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27.1.1  Test/Cycle  Overview 

Test/Cycle  supports  Computer  Power  Group’s  Testing  Management  Methodology.  The 
underlying  test  model  is  based  on  the  following  object  types: 

•  Project.  The  collection  of  all  data  associated  with  a  system  under  test. 

•  Requirement.  A  functional  specification  of  what  an  application  must  do. 

•  Build.  A  build  is  a  functionally  independent  group  of  modules  that  supports  a  well- 
defined  system  function  or  a  small  logical  subset  of  a  system. 

•  Test  plan.  Initially  records  the  overall  testing  strategy,  test  case  design,  and  test  case 
execution.  Subsequently  includes  information  on  the  builds,  test  runs,  and  test  cases 
associated  with  the  strategy,  and  records  the  test  execution  results. 

•  Test  run.  A  series  of  related  test  cases  combined  to  test  specific  requirements,  or  to 
perform  a  specific  category  of  testing. 

•  Test  case.  Consists  of  the  description  of  a  program’s  input  data  and  expected  output, 
together  with  a  description  of  the  steps  needed  to  execute  the  test  case. 

•  Test  file.  An  object  containing  information  about  the  actual  data  that  is  used  during 
execution  of  tests. 

•  Component.  A  product  of  the  development  process,  primarily  program  names,  source 
code  names,  and  libraries  supported  by  a  description  such  as  what  it  contains,  how  it 
is  created,  and  who  is  responsible  for  maintenance. 

Links  are  used  to  define  the  relationships  between  instances  of  these  objects.  While  each 
test  case  is  automatically  linked  to  a  single  test  run,  the  user  can  define  the  desired  links 
between  other  objects.  (The  link  between  test  runs  and  test  cases  is  the  only  non-commuta- 
tive  link;  it  is  established  from  the  test  case  perspective  so  that  every  test  case  must  belong 
to  a  test  run.)  Test/Cycle  does  impose  some  rules  that  guide  the  definition  of  legal  links;  for 
example,  only  leaf  requirements  may  be  linked  to  a  test  case. 

A  user  starts  by  defining  a  project  in  terms  of  a  unique  identifier,  organizational  infor¬ 
mation,  and  narrative  ASCII  text;  similar  information  is  captured  for  all  object  types.  The 
project  description  may  be  accompanied  by  characteristics  that  subsequently  can  be  used 
to  classify  both  requirements  and  test  cases. 

Typically,  the  next  step  is  to  define  the  requirements  that  will  be  used  to  drive  test  plan¬ 
ning.  An  initial  high-level  requirement  is  defined  and  successively  refined,  building  a  hier¬ 
archical  tree-like  structure  of  progressively  more  detailed  requirements.  Each  requirement 
is  automatically  assigned  a  hierarchy  level  number  that  indicates  its  relative  position  within 
the  hierarchy.  A  special  flag  is  used  to  indicate  whether  testing  of  a  requirement  is  required. 
A  status  of  testing  not  required  invokes  special  handling;  for  example,  the  user  must  give  a 
reason,  and  these  exceptional  cases  cannot  be  linked  to  a  test  case  or  a  test  run. 


PART  II 


Test/Cycle  &  Metrics  Manager 


The  user  then  proceeds  to  describe  the  other  necessary  objects.  In  addition  to  the  basic 
descriptive  information,  each  type  of  object  requires  some  special  data.  Builds,  for  exam¬ 
ple,  form  a  logical  grouping  of  test  runs  and  may  require  a  build  group  number  and  group 
sequence  number  to  indicate  build  sequencing.  Additionally,  a  test  level  attribute  represents 
the  test  level  (integration,  acceptance,  or  released)  achieved  by  a  build.  Test  runs  are  ac¬ 
companied  by  a  test  log  that  identifies  the  person  responsible  for  the  testing  and  maintains 
a  record  of  test  events,  and  test  run  sequencing  information  that  indicates  dependencies  be¬ 
tween  test  runs.  Test  cases  include  information  on  test  set  up,  the  tester,  and  one  or  more 
description  steps.  Each  test  case  description  step  includes  the  results  expected,  a  pass  or  fail 
indicator,  and  failure  action.  Test/Cycle  distinguishes  between  three  types  of  test  plan:  unit, 
integration,  acceptance.  In  each  case,  the  narrative  description  is  given  according  to  a  pre¬ 
formatted  outline,  and  is  accompanied  by  attributes  and  the  actual  test  plan  definition.  Test 
files  are  supported  by  capturing  the  record  type  for  records  contained  in  the  test  file.  Final¬ 
ly,  components  have  a  distinguishing  predefined  type  such  as  a  program  type. 

Based  on  its  placement  in  the  hierarchy  and  association  with  other  requirements,  a  re¬ 
quirement  may  be  subject  to  one  of  three  levels  of  testing:  high,  intermediate,  and  detail. 
Test/Cycle  generates  a  separate  validation  matrix  for  each  level.  In  the  High  Level  Valida¬ 
tion  Matrix  requirements  are  cross-referenced  to  builds,  in  the  Intermediate  Level  Valida¬ 
tion  Matrix  to  test  runs,  and  in  the  Detail  Level  Validation  Matrix  to  test  cases.  These 
matrices  show  both  explicit  and  implicit  links.  (An  implicit  link  is  one  created  when  a  re¬ 
quirement  in  the  subtree  of  a  higher  level  requirement  is  directly  linked  to  a  build,  test  run, 
or  test  case.)  One  of  their  primary  purposes  is  to  show  which  test  cases  test  which  require¬ 
ments  and  thus  provide  insight  into  test  completeness.  Test/Cycle  measures  requirements 
coverage  as  the  percentage  of  requirements  that  are  validated  by  a  set  of  test  cases.  Based 
on  user  entries,  Test/Cycle  also  tracks  the  number  of  times  a  test  run  is  executed.  This  al¬ 
lows  reporting  on  the  number  and  percent  of  test  steps  that  have  executed  successfully,  with 
the  date  and  time  a  test  case  was  100%  validated.  In  addition  to  ensuring  that  all  require¬ 
ments  are  tested  and  all  test  cases  are  used,  these  matrices  provide  additional  types  of  in¬ 
formation.  The  cross-reference  between  requirements  and  components,  for  example,  helps 
to  identify  the  parts  of  a  system  affected  by  a  requirement  change  and  the  tests  that  need  to 
be  rerun. 

Test/Cycle  provides  a  range  of  off-line  reports  and  several  on-line  reports.  Off-line  re¬ 
ports  are  available  to  provide  descriptions  of  the  existing  objects  of  each  type.  On-line  dis¬ 
plays  provide  various  status  information,  for  example,  the  status  of  each  test  case  linked  to 
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a  leaf  requirements  and  of  individual  test  cases  in  the  test  plan.  Additionally,  an  error/ex¬ 
ception  status  report  gives  a  series  of  eleven  consistency  checks  of  Test/Cycle  maintained 
data,  for  example,  test  cases  with  no  requirements  linked.  Some  of  these  checks  employ 
user-defined  threshold  values,  such  as  the  number  of  leaf  requirements  with  more  than  an 
acceptable  number  of  test  cases  linked.  Progress  reporting  is  provided  for  each  object.  In 
each  case,  an  on-line  validation  status  report  summarizes  the  status  of  test  cases  or  runs,  as 
appropriate,  linked  to  that  component.  This  information  includes  the  number  (and  percent¬ 
age)  of  test  cases,  or  runs,  that  have  passed,  and  cumulative  validation  statistics  for  the  ob¬ 
ject  in  question. 

Test/Cycle  reports  and  tracks  Work  Requests  (WRs).  These  can  be  classified  as  prob¬ 
lem ,  change,  or  other  (other  questions  or  discrepancies)  requests.  While  roughly  similar  in¬ 
formation  is  captured  for  problems  and  changes,  less  is  captured  for  other  requests.  For 
example,  statistical  information  is  only  kept  for  problems  and  changes;  details  are  kept  on 
problem  insertion  and  discovery,  a  characterization,  and  cost  to  fix  data  (for  problems)  or 
cost  to  implement  data  (for  changes).  Problem  and  change  requests  capture  information 
about  the  phase  and  activity  when  a  problem  was  introduced  and  when  identified;  this  pro¬ 
vides  some  support  for  continual  process  improvement.  They  allow  five  priority  levels, 
three  severity  levels,  and  distinguish  between  five  different  classifications. 

For  WRs  in  general,  sequence  numbers  are  automatically  assigned  to  support  an  audit 
trail.  WRs  are  treated  as  an  object  type  and,  consequently,  they  can  be  linked  to  instances 
of  any  other  object  type.  A  checklist  is  maintained  showing  the  status  of  each  WR.  Pre¬ 
defined  reports  are  available  to  provide  descriptions  of  individual  WRs  and  a  WR  log. 


27. 12  Metrics  Manager  Overview 

Metrics  Manager  supports  the  collection  and  analysis  of  quality  and  productivity  met¬ 
rics  for  management  purposes.  Data  can  be  collected  on  a  monthly,  quarterly,  or  annual  ba¬ 
sis  to  monitor  the  performance  of  an  organization  and  track  the  impact  of  new  methods, 
organizational  structures,  and  technologies.  The  user  starts  by  modeling  the  MIS  function. 
The  highest  level  of  structure  is  called  the  Enterprise.  An  Enterprise  consists  of  MIS  De¬ 
partments,  each  of  which  has  a  number  of  products.  For  metrics  reporting  and  graphing  pur¬ 
poses,  products  can  be  defined  as  members  of  an  overall  application  or  system,  yielding  a 
composite  product  called  an  Application.  Aggregates  are  a  special  subset  of  an  application 
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which  combine  products  that  are  not  part  of  a  single  application;  they  allow,  for  example, 
reporting  on  all  enhancements  regardless  of  product  identification. 

Data  is  stored  in  a  Basic  Operating  Database  (BOD)  which  captures  the  lowest  level 
data  for  each  enterprise,  each  MIS  department  within  the  enterprise,  and  for  each  product 
within  each  MIS  department.  This  includes  quality,  productivity,  cycle  time,  cost,  size, 
scope,  and  reliability  data  as  well  as  indicative  and  descriptive  data  used  for  comparison 
and  categorization  purposes.  Over  three  hundred  measures  and  attributes  are  captured. 
Once  a  set  of  basic  data  items  has  been  entered  in  the  BOD  and  validated,  the  quality  and 
productivity  metrics  can  be  computed.  These  are  computed  using  data  up  to  a  user-defined 
Period  Ending  Date,  and  only  for  products  whose  development  cycle  has  ended  by  this  date. 
Subsequently  database  reports  are  available  to  provide  information  at  the  enterprise,  MIS, 
and  product  levels. 

The  computed  data  can  also  be  used  to  produce  a  range  of  graphs  depicting  the  enter¬ 
prise  performance.  The  user  selects  x-  and  y-axes  from  sets  of  predefined  options  and,  op¬ 
tionally,  one  of  a  number  of  predefined  filters  that  select  a  subset  of  data  for  graphing.  Up 
to  eight  graph  definitions  can  be  stored  for  subsequent  reuse.  While  basic  metrics  graphs 
operate  against  enterprise  data,  the  user  can  request  graphs  that  compare  the  data  from  an 
Enterprise  to  the  data  from  the  industry  database.  (The  industry  database  is  a  regularly  up¬ 
dated,  statistically  validated  database  that  has  been  built  from  client  data.)  The  graphing 
function  also  allows  each  enterprise  to  define  a  Critical  Metric  Set  (CMS),  that  is,  those 
metrics  that  have  been  determined  to  be  key  management  factors  in  determining  enterprise 
success.  Graphs  are  accompanied  by  a  graph  audit  report  that  prints  the  values  from  each 
product  that  are  used  to  develop  each  graph.  This  is  intended  for  use  in  understanding  the 
metrics  calculations  and  in  comparing  the  same  attribute  or  metric  across  all  applicable 
products.  Graphs  are  displayed  on  the  screen,  but  may  be  sent  to  a  file  for  subsequent  print¬ 
ing  on  a  dot-matrix  or  laser  printer. 

Finally,  a  merge/extract  function  is  provided  for  those  cases  where  multiple  copies  of 
Metrics  Manager  are  installed  across  an  organization.  It  is  used  to  consolidate  data  into  a 
single  database  for  analysis  and  reporting  purposes. 
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27.2  Observations 

Ease  of  use.  These  tools  use  an  object-oriented,  graphical.  Common  User  Access 
(CUA)  user  interface  to  facilitate  tool  use.  Both  interfaces  are  menu  driven  with  context- 
sensitive  on-line  help  and  an  analog  box  of  information  for  selections  where  appropriate. 

The  user  can  use  a  mouse  to  select  objects  in  different  ways:  “point  and  click,”  double 
click,  or,  unique  to  Test/Cycle,  “drag  and  drop”  which  represents  direct  manipulation  of  ob¬ 
jects;  accelerator  keys  are  provided  as  an  alternative  method  of  user  interaction.  Two  types 
of  editing  are  available.  The  first  is  a  limited  edit  using  keyboard  editing  keys  such  as  insert, 
backspace,  home.  The  second  type  available  through  the  zoom  narrative  button  is  a  full  edit 
which  allows  use  of  the  Windows  Clipboard  Editing  facility.  The  search  filters  and  user- 
tailorable  project  characteristics  provide  a  basis  for  on-line  searching  of  the  test  data  base. 
Test/Cycle  also  supports  on-line  browsing  of  the  links  between  objects.  Test/Cycle  sup¬ 
ports  only  limited  customization:  the  test  plan  format  can  be  modified  by  editing  the  test 
plan  narrative  file  to  include  a  desired  outline. 

Documentation  and  user  support.  The  documentation  for  these  tools  is  easy  to  follow 
and  complete. 

Problems  encountered.  No  problems  were  encountered  in  the  use  of  these  tools. 

27.3  Planned  Additions 

A  new  version  of  Metrics  Manager  is  due  for  release  in  late  1992.  This  new  release  will 
include  additional  functionality.  For  example.  Metrics  Manager  will  help  a  user  to  deter¬ 
mine  the  scope  of  impact  of  a  proposed  requirements  change,  and  allow  reporting  on  this 
maintenance  effort  independently  from  the  rest  of  the  system.  Both  Test/Cycle  and  Metrics 
Manager  will  be  integrated  with  ABT  Technologies  Project  Manager  Workbench  and  mar¬ 
keted  by  ABT  Technologies.  In  October  1992,  Computer  Power  Group  will  release  a  ver¬ 
sion  of  Metrics  Manager  that  integrates  with  Test/Cycle. 

Future  versions  of  Test/Cycle  will  incorporate  Microsoft  Word  to  support  entry  and  ed¬ 
iting  of  requirements.  Additionally,  user-defined  reporting  and  reliability  analysis  will  be 
supported. 
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27.4  Sample  Outputs 

Figures  27- 1  through  27-22  provide  sample  outputs  from  Test/Cycle  and  Metrics  Man¬ 
ager. 
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Page  1 

Project:  Gift  Pack  Order  System 
Wednesday,  September  16,  1992 
11:10:29 

TEST/CYCLE  REQUIREMENTS  HIERARCHY  REPORT 

Ancestors : 


None. 

Parent: 


Gift  Pack  Order  System 
Children : 


1  System  Entry  (Main  Window)  + 

2  Maintain  Order  File  + 

3  Order  Inquiries  + 

4  Work  Order  Processing  + 

5  Order  Piok-up/Delivery  + 

6  End-of-Day  Processing  + 

7  Report  Generation  + 


Page  2 

TEST/CYCLE  REQUIREMENTS  HIERARCHY  REPORT 

Ancestors : 

Gift  Pack  Order  System 
Parent: 


1  System  Entry  (Main  Window) 
Children : 


1.1  Menu  Selection  + 

1 . 2  Maintain  User  IDs 

1.3  Maintain  Statistics 

1.4  Maintain  Parameter  Values 
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Ancestors : 


Figure  27-1.  Test/Cycle  Requirements  Hierarchy  Report 
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Gift  Pack  Older  Systen 
1  System  Entry  (Mein  Window) 

Parent: 

1.1  Menu  Selection 
Children : 


1.1.1 

1.1.3 

1.1.3 

1.1.4 

1.1.5 

1.1.6 

1.1.7 

1.1.8 

1.1.9 

1.1.10 
1.1.11 
1.1.13 

1.1.13 

1.1.14 

1.1.15 

1.1.16 


Enter  GPOS  System 
Format  Main  Menu 
Validate  Menu  Selection 
Validate  User  ZD 
Xfer  to  Order  Entry  Scr 
Xfer  to  Work  ords  Scr 
Xfer  to  Ord  Pick-up  Scr 
Xfer  to  End  of  Day  Proo  Scr 
Xfer  to  Ord  Maint  Sor 
Exit  System 

Disp  *110  Invalid  User  ZD*  Msg  - 
Disp  *130  Znv  Select*  Err  Msg  - 
Disp  *0rd  xxxxxx  Entered*  Msg  - 
Xfer  to  Inquiry  Scr 
Xfer  to  Reports  Scr 
Xfer  to  Par an  File  Maint 
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Anoestors i 

Gift  Pack  Order  System 
7  Report  Generation 

Parent :  > 

7.1  Report  Menu 
Children: 


1.3  Maintain  User  ZDs 

7.1.1  Generate  Analysis  Report 

7.1.3  Print  Work  Orders  on  Request  - 


Figure  27*1  continued:  Test/Cycle  Requirements  Hierarchy  Report 
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Project:  Gift  Pack  Order  System 
Wednesday/  September  16,  1992 
11:05:36 


TEST/CYCLE  REQUIREMENT  DESCRIPTION  REPORT 

Requirement  ID:  1.1  Menu  Selection 

Author:  Jackie  Jones 


Requestor: 

Testing  Not  Required:  Not  Applicable 
Narrative : 

No  Narrative  Found 
Linkages : 


To  Characteristics; 

None. 

To  Builds  .- 
None. 

To  Test  Cases: 

None. 

To  Test  Runs: 

Main  Menu  Entries  Validation 

To  Components : 

None. 

To  Test  Plans: 

None. 

To  Work  Requests: 

None. 


Figure  27*2.  Test/Cycle  Requirement  Description  Report 
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Figure  27*4.  Test/Cycle  Intermediate  Level  Matrix  Screen 


Figure  27-5.  Test/Cycle  Detail  Level  Matrix  Screen 


27-11 


Test/Cycle  &  Metrics  Manager 


PART  il 


Page  1 

Project:  Gift  Pack  Order  System 
Wednesday/  September  16,  1992 
11:24:27 


TEST/CYCLE  BUILD  DESCRIPTION  REPORT 

Build  ID:  1-Systea  Entry  t  Par ax  Maint 
Author :  Jackie  Jones 

Narrative : 

This  build  consists  of  the  main  menu  and  parameter  file  maintenance 
functions.  These  functions  allow  the  user  to  customise  the  system.  The 
main  menu  function  allows  the  user  to  select  one  of  nine  functions:  Order 
Entry,  Work  Order  Processing,  Order  Pick-Up,  End- of -Day  Processing,  Order 
Maintenance,  Inquiry,  Report  Generation,  Parameter  File  Maintenance,  or 
System  Exit,  When  the  user  selects  the  exit  function 

Build  Group  Number: 

Sequence  Number: 

Test  Level  Achieved:  Integration  Demoted:  No 

Linkages : 

To  Test  Runs: 

None. 

To  Requirements: 

System  Entry  (Main  Window)  Update  Ord  Status  to  Complete 

To  Components : 

None. 

To  Test  Plans: 

None. 

To  work  Requests: 

None. 


Figure  27-6.  Test/Cycle  Build  Description  Report 


27-12 


PART  II 


Test/Cycle  &  Metrics  Manager 


Page  2 

Pro j act i  Gift  Pack  Order  System 
Wednesday/  September  16,  1992 
Ili31>44 

TEST/CYCLE  COMPONENTS  DESCRIPTION  REPORT 

Component  ID:  Programl 
Author :  Jackie  Jones 

Narrative : 

This  is  a  sample  of  a  program  description.  It  is  used  to  — 

Kind  of  Component:  Program 
Linkages : 


To  Requirements: 
Order  Enquiries 

To  Builds: 

None. 

To  Test  Cases: 
None. 

To  Test  Files: 
None. 

To  Test  Plans: 
None. 

To  Work  Requests: 
None. 


Figure  27*7.  Test/Cycle  Components  Description  Report 
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Project:  Gift  Pack  Order  System 
Wednesday,  September  16,  1992 
11:28:22 


TEST/CYCLE  TEST  RUM  DESCRIPTION  REPORT 

Test  Run  ID:  MM  Select  Validation  ^ 

Author :  Jackie  Jones 

Narrative : 

The  Maim  Menu  Select  Validation  Test  Run  consists  of  a  set  of  test  cases 

that  attempt  to  ■break"  the  selection  criteria  of  the  main  menu.  In  # 

addition  to  verifying  that  the  standard  selection  methods  work,  the  test 
cases ,  which  are  a  capture/playback  of  screen  Images ,  try  to  . . . 


Test  Run  Group  Number: 

Test  Run  Sequence: 

Test  Log: 

Tom  Smith  7/25/91 


Linkages : 


To  Builds: 

None.  # 

To  Requirements: 

None. 

To  Test  Cases; 

None. 

To  Test  Files:  0 

None. 

To  Test  Plans: 

None. 

To  Work  Requests: 

None. 


Figure  27-8.  Test/Cycle  Test  Run  Description  Report 
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Figure  27-9.  Test/Cycle  Requirements  Validation  Status  Screen 
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Figure  27-10.  Test/Cycle  Test  Run  Validation  Status  Screen 
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Projects  Gift  Pack  Order  Syeteai 
Wednesday/  September  16,  1993 
Ili40sl8 


TEST/CYCLE  TEST  CASE  DESCRIPTION  REPORT 

Teat  Case  IDs  Inv  IDs 

Author  s  Jackie  Jones 
Tester  s 

Setup  for  Tests 


System  Started,  Main  Menu  on  Screen,  cursor  on  User  ID 
field 

Test  Steps: 


Pass  Description  of  Test  Expected  Results 
1.  Yes  Hit  < ENTER) 


Action  if  Fail 


-Display  Err Mag  *110 
Invalid  User  ID" 

-Cursor  on  User  ID. 


-User  ID 
Highlighted 

2.  Yes  -  Key  "123"  in  User  -Display  Err Msg 

ID.  "110  Invalid  User 

ID." 

Hit  < ENTER)  -User  ID 

Highlighted 


Linkages : 

To  Characteristics: 

None. 

To  Requirements: 

Disp  "110  Invalid  User  ID*  Msg 
Display  Ord  Inf  for  Pick-up 

Narrative : 


Format  Main  Menu 
Validate  Menu  Selection 


The  Parameter  File  must  contain  one  or  more  valid  user  IDs  for  validation 
of  the  Main  Menu  User  ID  and  Selection  entries. 

Figure  27-11.  Test/Cycle  Test  Case  Description  Report 
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Figure  27-13.  Test/Cycle  Test  Case  Referenced  by  Requirement  Screen 
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Project:  Gift  Pack  Order  Systaa 
Wednesday/  Sept enter  16,  1992 
11:36:03 


TEST /CYCLE  TEST  FILE  DESCRIPTION  REPORT 

Test  File  ID:  Main  Menu  Val  IDs  Par am  File 
Author:  Jackie  Jones 

Narrative ; 

The  Parameter  File  must  contain  one  or  more  valid  user  IDs  for  validation 
of  the  Main  Menu  User  ID  and  Selection  entries. 

Test  File  Type:  Sequential 

Linkages : 

To  Test  Runs: 

None. 

To  Test  Cases: 

Nona. 

To  Components: 

None. 

To  MOrk  Requests: 

None. 


Figure  27-14.  Tost/Cycle  Test  File  Description  Report 
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10 s  i  Category i  Open  Problem 

Instigator:  Jackie  Jones 
Requestor  t 

HR  Dates  1/8/92  Priority: 

WR  Types 
Narrative : 

This  is  a  sample  work  request  for  a  problem  report.  Zt  is  used  to  . . . 
Status:  None 

Error:  No  Problem  metrics  record  found  for  1 
Linkages : 

Paradox  ISAM  Emulator (Cmp) . 


Figure  27-1 S.  Test/Cycle  Work  Request  Description 


Project:  Gift  Pack  Order  System 
Wednesday,  September  16,  1992 
11:44:10 

TEST/CTCLE  WORK  REQUEST  LOG  REPORT 


Open  Problem  Work  Requests 


Cat  ID 


OP 

OP 


1 

2 


Priority  Type 


Narrative 


<some  text> 
<some  text> 


Status 


None 

None 


Figure  27-16.  Test/Cycle  Work  Request  Log  Report 


t 
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SAMPLE  OUTPUTS  FROM  MARS 

09/24/92  llt20  METRICS  MANAGER 

[MDB001]  V3.02  METRICS  DATA  BASE 

Enterprise:  5550 

PERIOD  ENDING  SEP,  1992 


Page 


1 


ENTERPRISE 


TOTAL  SALES  REVENUE: 

TOTAL  NUMBER  OP  EMPLOYEES: 
SIC  CODE: 

SECONDARY  SIC  CODE: 


$700,000,000 

6,000 


-1 

-1 


Figure  27-17.  Metrics  Manager  Database  Full  Report 
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METRICS  MANAGER 
METRICS  DATA  BASE 
Enterprise:  5550 

MIS:  Management  Information  Systems 
PERIOD  ENDING  SEP,  1992 


MIS 


HARDWARE  BUDGET: 

$2,000,000 

SOFTWARE  BUDGET: 

$500,000 

PERSONNEL  BUDGET: 

$7,000,000 

FACILITIES  BUDGET: 

$2,000,000 

OTHER  BUDGET: 

$2,000,000 

TOTAL  BUDGET: 

$13,500,000 

TOTAL  FUNCTION  POINTS 

OF  INSTALLED  BASE: 

0 

NUMBER  OF  TERMINALS  USED  FOR 
DEVELOPMENT  AND  MAINTENANCE  — 
Mainframe  Terminals: 

150 

PC  Workstations-. 

100 

TOTAL  MIPS: 

46.1 

TOTAL  PROGRAMS: 

4,582 

TOTAL  KLOC: 

2,726 

TOTAL  STAFF: 

315 

Total  Maintenance  Staff: 

-1 

Total  Technical  Staff: 

85 

\ 

AVERAGE  DEPARTMENT  LABOR  RATE: 

$25.00 

MAINTENANCE  LABOR  RATE: 

$-1.00 

09/24/92  11:20 

(MDB002]  V3.02 


Page  2 


Figure  27*17  continued:  Metrics  Manager  Database  Full  Report 
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09/24/92  Hi 20  METRICS  MANAGER  Page 

[MDB003]  V3.02  METRICS  DATA  BASE 

Enterprise:  5550 

MIS  i  Manageaent  information  Systems 
Product:  Product  A  -  Development 
PERIOD  ENDING  SEP,  1992 

PRODUCT  ATTRIBUTES 


TYPE  OF  EFrORT:  A  -  New  Development 


DEVELOPMENT  END  DATE:  04/88 

DATE  OF  ORIGINAL  REQUEST:  /  / 

DATE  NEEDED:  02/27/87 

DATE  STARTED:  05/31/86 

DATE  OF  FIRST  PRODUCTION  UTILIZATION:  04/30/88 

DATE  LAST  COMPONENT  INSTALLED:  04/30/88 

DURATION:  700  DAYS 

BACKLOG  PERIOD:  0  DAYS 


TARGET  DATES 

Original:  01/01/88  Revised:  /  / 


Actual:  01/01/88 


DAYS  VARIANCE: 


0  DAYS 


TOTAL  KLOC: 


29 


FUNCTION  POINT  COUNTS  (Actual  Approved) 


Adjusted  Function  Points  (IFPUG) :  1,630 

Function  Points  Override:  0 

USER  INVOLVEMENT 

Definition  Phase:  H 

Construction  Phase:  H 

Operation  Phase:  H 

PERCENT  OF  MIS  TO  TOTAL  PERSONNEL 

Definition  Phase:  90% 

Construction  Phase:  90% 

Operation  Phase:  95% 


PROJECT  MANAGER'S  EXPERIENCE: 
STAFF  APPLICATION  EXPERIENCE: 
TECHNICAL  YEARS  OF  EXPERIENCE: 


2  Years 
2  Years 
6  Years 


SYSTEM  AVAILABILITY: 
SYSTEM  RESPONSE  TIME: 


95% 

5  Seconds 


NET  PRESENT  VALUE:  -1 

RETURN  ON  INVESTMENT:  -1.00% 


ACTUAL  PEAK  TEAM  SIZE: 


5 


Figure  27-17  continued:  Metrics  Manager  Database  Full  Report 
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09/34/92  11 i 20  METRICS  MANAGER  Page  4 

[MDB004J  V3.02  METRICS  DATA  BASE 

Enterprise :  5550 

MIS i  Management  Information  Systems 
Product:  Product  A  -  Development 

PERIOD  ENDING  SEP,  1992 

I 

PRODUCT  COST 


DEVELOPMENT  COSTS 

Original 

Revised 

Actual 

Labor 

$350,868 

$575,984 

$737,523 

Hardware 

$70,000 

$120,296 

$263,153 

Expense 

$0 

$0 

$0 

TOTAL 

$420,868 

$696,280 

$1,000,676 

ACTUAL  TECHNICAL  LABOR  COST: 


$0 


TOTAL  PRODUCTION  COST: 
Production  Rerun  Cost: 


$10,400 

$-1 


DEFECT  REMOVAL 

Definition 

Construction 

Operation 

COSTS 

Phase 

Phase 

Phase 

TOTAL 

Labor 

$2,600 

$14,325 

$33,575 

$50,500 

Machine 

$-1 

$5,900 

$14,800 

$20,700 

TOTAL 

$2,600 

$20,225 

$48,375 

$71,200 

FAILURE  COSTS 

4 

Internal  Labor: 

$25,900 

Internal  Machine: 

$1,000 

Total  Internal  Failure: 

$26,900 

External  Failure: 

$-1 

Production  Rerun  Cost: 

$-1 

TOTAL  FAILURE  COST: 

$26,900 

Figure  27-17  continued:  Metrics  Manager  Database  Full  Report 


27-23 


> 


Test/Cycle  &  Metrics  Manager 


PART  II 


09/24/92  11:20  METRICS  MANAGER 

[MDB005]  V3.02  METRICS  DATA  BASE 

Enterprise:  5550 

MIS:  Management  Information  Systems 
Product:  Product  A  -  Development 

PERIOD  ENDING  SEP,  1992 


PRODUCT  EFFORT  (HOURS) 


DEVELOPMENT  LABOR  EFFORT 


Original : 

8,354 

HOURS 

Revised : 

12,058 

HOURS 

Actual: 

15,365 

HOURS 

ACTUAL  TECHNICAL 

LABOR  EFFORT: 

0 

HOURS 

DEFECT  REMOVAL  EFFORT 

Definition  Phase: 

104 

HOURS 

Construction  Phase: 

573 

HOURS 

Operation  Phase: 

1,343 

HOURS 

TOTAL: 

2,020 

HOURS 

INTERNAL  FAILURE  EFFORT: 

1,036 

HOURS 

Page 


Figure  27*17  continued:  Metrics  Manager  Database  Full  Report 
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METRICS  MANAGER  Page  6 

METRICS  DATA  BASE 
Enterprise:  5550 

MIS:  Management  Information  Systems 
Product:  Product  A  -  Development 

PERIOD  ENDING  SEP,  1992 


PRODUCT  QUALITY 


DEFECTS  BY 


INSERTION  PHASE 

Minor 

Moderate 

Severe 

Total 

In  Definition 

-1 

85 

-1 

85 

In  Construction 

-1 

66 

-1 

66 

In  Operation 

-1 

167 

-1 

167 

TOTAL 

0 

318 

0 

318 

DEFECTS  BY 

DETECTION  PHASE 

Minor 

Moderate 

Severe 

Total 

Definition 

-1 

26 

-1 

26 

Construction 

-1 

125 

-1 

125 

Operation 

-1 

167 

-1 

167 

TOTAL 

0 

318 

0 

318 

Minor 

t- 

Moderate 

Severe 

Total 

FAILURES 

-1 

126 

-1 

126 

09/24/92  11:20 

[MDB006]  V3.02 


I 


I 


i 

Figure  27-17  continued:  Metrics  Manager  Database  Full  Report 
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METRICS  MANAGER  Pag*  7 

METRICS  DATA  BASE 
Enterpxi.se:  5550 

MIS:  Management  Information  Systems 
Product:  Product  A  -  Development 
PERIOD  ENDING  SEP,  1992 

PRODUCT  METRICS 

MONTHS  IN  OPERATION:  12  Months 

PRODUCTIVITY 

KLOC  Per  Staff  Month:  0.327 

Function  Points  Per  Staff  Month:  18.388 

CYCLE  TIME  (Elapsed  Days  /  Function  Point) 

Overall  Cycle  Time:  0.000  Days 

Development  Cycle  Time:  0.429  Days 

COST  /  FUNCTION  POINT 
Development  unit  Cost: 

Development  Labor  Unit  Cost: 

Development  Defect  Cost  Rate: 

COST  PER  MONTH  /  FUNCTION  POINT 
Production  Defect  Cost  Rate: 

Failure  Cost  Rate: 

Production  Unit  Cost  Rate; 

RELIABILITY 

Mean  Time  To  Failure: 

Monthly  Failure  Rate: 

Monthly  Failure  Rate  1st  6  Months: 

Monthly  Failure  Rate  Last  6  Months: 

Failures  Per  Execution  Hour: 

Failure  Rate  Per  Execution  Hour: 

PERFORMANCE 

Effort  Variance: 

Cost  variance: 

Schedule  Variance: 

QUALITY 

Defect  Density: 

Development  Defect  Removal  Efficiency 
(Development  Defects  /  All  Defects): 


CUSTOMER  SURVEY  METRICS: 

User  Satisfaction  Strategic  Tactical 
With  SDP  Value  Value 

8804 :  N/A 


0.195  Defects  /  FP 
0.475 


27.00* 

44.00* 

0.00* 


0 . 0952  Months  per  Failure 
0.0064  Failures  /  Month  /  FP 
0.0090  Failures  /  Month  /  FP 
0.0039  Failures  /  Month  /  FP 
9 . 0000  Failures  per  Hr 
0.0055  Failures  per  Hr  /  FP 


$2.47 

$1.38 

$0.53 


$613.91 

$452.40 

$14.00 


09/24/92  11:20 

[MDB007 ]  V3.02 


Figure  27-17  continued:  Metrics  Manager  Database  Full  Report 
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09/34/92  11:20  METRICS  MANAGER  Page  8 

[MDB003]  V3.02  METRICS  DATA  BASE 

Enterprise:  5550 

MIS:  Management  Information  systems 
Product:  Product  B  -  1st  Enhancement 

i  PERIOD  ENDING  SEP,  1992 


PRODUCT  ATTRIBUTES 


> 


I 


I 


TYPE  OF  EFFORT:  B  -  Major  Enhancement 


DEVELOPMENT  END  DATE:  01/89 

DATE  OF  ORIGINAL  REQUEST:  09/01/87 
DATE  NEEDED:  01/01/89 
DATE  STARTED:  06/01/87 
DATE  OF  FIRST  PRODUCTION  UTILIZATION:  01/01/89 
DATE  LAST  COMPONENT  INSTALLED:  01/31/89 
DURATION:  610  DAYS 
BACKLOG  PERIOD:  -92  DAYS 


TARGET  DATES 

Original:  10/01/88  Revised:  /  /  Actual:  01/31/89 

DAYS  VARIANCE:  122  DAYS 


TOTAL  KLOC: 


1 


I 


» 


► 


Figure  27*17  continued:  Metrics  Manager  Database  Full  Report 
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09/25/92  09 i 52 

[MDB008]  V3.02 

PERIOD  ENDING  SEP,  1992 
MIS  SUMMARY 

Total  products  in  MIS:  5 


METRICS  MANAGER  Page  1 

METRICS  DATA  BASE 
Enterprise:  5550 

MIS:  Management  Information  Systems 


Basic  Measures - Metrics - 


KLOC  PER  STAFF  MONTH:  1.027 

TOTAL  KLOC:  314  F/P  PER  STAFF  MONTH:  18.388 

TOTAL  FUNCTION  POINTS:  1,630  Cost  /  Function  Point 

DEVELOPMENT  UNIT  COST:  $613.91 

ACTUAL  LABOR  EFFORT:  52,981  DEVEL.  LABOR  UNIT  COST:  $452.47 

ACTUAL  DEVELOPMENT  COST:  $2,874,715  DEV.  DEFECT  COST  RATE:  $14.00 

TOTAL  DEFECT  REMOVAL  COST:  $82,500  PROD.  DEFECT  COST  RATE:  $2.47 

TOTAL  FAILURE  COST:  $32,075  FAILURE  COST  RATE:  $1.38 

PROD.  UNIT  COST  RATE:  $0.53 

TOTAL  DEFINITION  DEFECTS:  50  Quality 

TOTAL  CONSTRUCTION  DEFECTS:  152  - 

TOTAL  OPERATION  DEFECTS:  175  DEFECT  DENSITY  PER  F/P:  0.1951 

-  DEVEL.  DEFECT  REMOVAL 

TOTAL  DEFECTS:  377  EFFICIENCY:  0.536 

TOTAL  DEFECTS  INSERTED  IN  —  Reliability 

DEFINITION:  111  - 

CONSTRUCTION:  91  FAILURES  PER  EXECUTION  HOUR:  2.3276 

OPERATION:  175 


TOTAL  FAILURES:  135  , 


Figure  27-18.  Metrics  Manager  Enterprise  &  MIS  Metric  Summary  Report 
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09/25/92  09:52  METRICS  MANAGER  Page  2 

[MDB008]  V3.02  METRICS  DATA  BASE 

Enterprise:  5550 


PERIOD  ENDING  SEP,  1992 
ENTERPRISE  SUMMARY 
Total  products  in  ENTERPRISE:  5 

Basic  Measures - Metrics' 


TOTAL  KLOC: 

314 

TOTAL  FUNCTION  POINTS: 

1,630 

ACTUAL  LABOR  EFFORT: 

52,981 

ACTUAL  DEVELOPMENT  COST:  $2, 

874,715 

TOTAL  DEFECT  REMOVAL  COST: 

$82,500 

TOTAL  FAILURE  COST: 

$32,075 

TOTAL  DEFINITION  DEFECTS: 

50 

TOTAL  CONSTRUCTION  DEFECTS: 

152 

TOTAL  OPERATION  DEFECTS: 

175 

TOTAL  DEFECTS: 

377 

TOTAL  DEFECTS  INSERTED  IN  — 

DEFINITION: 

111 

CONSTRUCTION: 

91 

OPERATION: 

175 

TOTAL  FAILURES: 

135 

KLOC  PER  STAFF  MONTH:  1.027 

F/P  PER  STAFF  MONTH:  18.388 

Cost  /  Function  Point 

DEVELOPMENT  UNIT  COST:  $613.91 

DEVEL.  LABOR  UNIT  COST:  $452.47 

DEV.  DEFECT  COST  RATE:  $14.00 

PROD.  DEFECT  COST  RATE:  $2.47 

FAILURE  COST  RATE:  $1.38 

PROD.  UNIT  COST  RATE:  $0.53 

Quality 

DEFECT  DENSITY  PER  F/P:  0.1951 

DEVEL.  DEFECT  REMOVAL 

EFFICIENCY:  0.536 

Reliability 


FAILURES  PER  EXECUTION  HOUR:  2.3276 


Figure  27-18  continued:  Metrics  Manager  Enterprise  &  MIS  Metric  Summary  Report 
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Figure  27-19.  Metrics  Manager  Function  Points  Productivity  vs.  Type  of  Effort 
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Figure  27-20.  Metrics  Manager  Development  Detect  Removal  Efficiency  vs.  Size  of  Product 
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Figure  27-21 .  Metrics  Manager  Development  Defect  Removal  Efficiency  vs.Tools  Used 
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Figure  27-22.  Metrics  Manager  Development  Unit  Cost  vs.  Size  of  Product  Showing  Industry 

Data 
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28.  TestGen,  QualGen,  GrafBrowse,  and  the  ADADL  Processor 

TestGen,  QualGen,  GrafBrowse,  and  the  ADADL  processor  are  part  of  the  AISLE  fam¬ 
ily  of  software  tools  based  on  the  Ada-based  Design  and  Documentation  Language 
(ADADL).  ADADL  itself  is  fully  compilable  by  any  Ada  compiler  and  has  been  selected 
by  the  Joint  Integrated  Avionics  Working  Group  (JLAWG)  as  the  Ada  program  design  lan¬ 
guage  (PDL)  to  use  for  the  Army’s  Light  Helicopter  (LH)  and  the  Air  Force’s  Advanced 
Tactical  Fighter  (ATF)  programs. 

The  ADADL  processor  provides  static  analysis  of  ADADL  designs  and  Ada  code,  and 
produces  the  outputs  needed  by  TestGen,  QualGen,  and  GrafBrowse.  These  other  tools  also 
operate  on  both  ADADL  designs  and  Ada  code.  TestGen  supports  the  review  of  ADADL 
designs,  preparation  of  unit  test  plans,  and  test  coverage  analysis.  QualGen  reports  on  qual¬ 
ity,  using  a  user-tailorable  metrics  hierarchy.  GrafBrowse  supports  code  browsing  and  cre¬ 
ates  invocation  trees  to  assist  in  reverse  engineering. 

Other  AISLE  tools  include  syntax-directed  and  graphical  editors,  an  automatic  docu¬ 
ment  generator,  a  design  database  analyzer,  a  requirements  analyzer  and  tracer,  and  a  com¬ 
pilation  order  analyzer. 


28.1  Tool  Overview 

The  AISLE  tool  family  is  marketed  by  Software  Systems  Design.  It  has  been  available 
since  1984  and  has  over  1,000  users.  The  tools  are  available  on  a  wide  range  of  machines 
such  as  VAX,  VAXStation,  and  MicroVAX  under  VMS,  Unix,  or  Ultrix;  and  Sun-3  and 
Sun-4,  HP9000-800,  Apollo,  DecStation,  and  80386-based  PC  systems  under  MS-DOS  or 
Unix.  Where  windowing  is  required,  the  tools  support  X-Windows,  OpenWindows,  Sun- 
Windows,  DECwindows,  Tektronix,  and  Hewlett-Packard  windows.  Graphics  output  is 
formatted  for  a  range  of  devices.  These  formats  include  Postscript,  Tektronix,  and  Graphi¬ 
cal  Kernel  System  (GKS).  AISLE  can  interface  to  the  Teamwork,  StP,  and  Excelerator 
CASE  systems  to  provide  automatic  generation  of  designs  from  requirements.  At  the  time 
of  examination,  TestGen  prices  started  at  $4,600,  the  ADADL  processor  at  $5,000,  Qual¬ 
Gen  at  $4,000,  and  GrafBrowse  at  $5,500.  Training  and  consulting  services  are  available. 

The  IDA  study  used  the  ADADL  processor  version  5.3.E,  TestGen  version  2.2.2,  Qual¬ 
Gen  version  1.1,  and  GrafBrowse  version  2.2.2  running  on  a  Sun-4  system  under  Open- 
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Windows.  The  examinations  focused  on  application  of  the  tools  to  Ada  code  rather  than  to 
ADADL  designs. 


28.1.1  ADADL  Processor  Overview 

The  user  starts  the  application  of  these  tools  by  submitting  the  code  to  the  ADADL  pro¬ 
cessor.  When  the  code  exists  in  several  files,  these  must  be  submitted  in  compilation  order. 
The  processor  itself  consists  of  over  25  tools,  although  these  are  transparent  to  the  user. 
While  some  of  these  tools  operate  only  on  ADADL  designs,  the  majority  apply  to  either 
ADADL  designs  and  Ada  code,  or  just  to  Ada  code.  This  latter  category  of  tools  includes 
the  following: 

•  A  pretty  printer. 

•  An  object  highlighter  to  highlight  use  of  Ada  entities. 

•  A  program  unit  invocation  tree  generator. 

•  A  program  unit  declaration  hierarchy  generator. 

•  Several  cross  references,  including  an  object,  type,  subtype  and  derived  type  parent 
reference  cross  references. 

•  Generic  instantiation  report  shows  the  location  of  each  generic  instantiation. 

•  Interrupt  report  generator  shows  how  interrupts  are  declared  and  where  they  are  used. 

•  With  hierarchy  generator  shows  the  library  units  imported  by  each  program  unit. 

•  Quality  analyzer  identifies  design  or  code  portions  that  do  not  conform  to  quality 
guidelines,  for  example,  objects  declared  but  not  used,  and  program  unis  with  an 
ADADL  complexity  greater  than  some  maximum  limit 

•  Complexity  analyzer  computes  the  cyclomatic  complexity  and  ADADL  complexity 
measures. 

•  Undefined  identifier/spelling  checker  identifies  possible  spelling  errors  or  the  omis¬ 
sion  of  a  declaration. 

Report  formatting  can  be  modified,  or  the  production  of  specific  reports  disabled,  using 
ADADL  processor  commands.  Like  ADADL  design  statemens,  these  are  given  in  the 
form  of  special  Ada  commens.  Additional  special  commens  include  the  following: 

•  Data  dictionary  definitions.  Used  to  produce  a  data  dictionary  of  all  Ada  program 
unis,  types,  and  objecs. 

•  Project  management  information.  Used  to  define  information  useful  to  the  project 
manager,  for  example,  dates  of  completion  of  design,  coding,  and  testing  activities. 

The  use  of  these  special  commens  was  not  examined. 
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28.1.2  TestGen  Overview 

TestGen  supports  structural  testing  at  three  levels:  branch  testing,  structured  testing 
based  on  McCabe’s  cyclomatic  complexity  number  (also  called  basis  path  testing)  [Mc¬ 
Cabe  1976],  and  path  testing.  Using  a  selected  technique,  TestGen  gives  the  number  of  nec¬ 
essary  test  cases  and  then,  for  each  test  case,  identifies  the  program  conditions  required  at 
each  decision  point  to  exercise  the  necessary  program  paths  and  shows  the  statements  that 
will  be  executed  during  that  test.  This  information  helps  the  user  derive  necessary  test  data 
for  structural  testing.  Since  each  level  of  testing  may  require  a  potentially  very  large  num¬ 
ber  of  test  cases,  TestGen  provides  an  option  that  allows  the  user  to  first  see  how  many  test 
cases  will  be  required  for  each  program  unit  using  the  different  testing  techniques.  This  in¬ 
formation  can  be  used  to  guide  subsequent  test  case  generation;  it  is  also  useful  in  estimat¬ 
ing  testing  costs  in  terms  of  the  number  of  test  cases  required  for  structural  analysis. 

Before  instrumenting  code  :or  coverage  data  collection,  the  user  must  establish  a 
TestGen  library.  Special  utinties  are  provided  for  such  library  creation  and  maintenance. 
Once  an  appropriate  library  has  been  created  and  opened,  the  user  specifies  the  files  that 
should  be  instrumented.  The  user  can  further  limit  the  extent  of  instrumentation  by  request¬ 
ing  instrumentation  of  selected  program  units  instead  of  all  the  units  in  the  file.  The  instru¬ 
mentation  process  also  requires  the  creation  of  a  simple  test  driver.  If  the  program  under 
test  contains  a  main  procedure,  TestGen  can  automatically  generate  the  necessary  driver. 
(This  driver  performs  a  loop  calling  the  instrumented  program  for  as  long  as  the  user  wish¬ 
es.)  Otherwise,  a  special  test  driver  must  be  manually  created  by  the  user  based  on  a  tem¬ 
plate  supplied  in  the  documentation. 

The  user  manually  compiles  and  links  the  instrumented  code  and  test  driver.  When  in¬ 
voked,  the  test  driver  queries  the  user  for  a  run  identification  and  brief  description.  As  the 
program  executes  on  test  data,  the  instrumentation  produces  a  trace  history  that  records  the 
order  in  which  statements  were  executed.  (The  run  identification  is  used  in  naming  the  gen¬ 
erated  trace  file.)  The  Test  Coverage  Analyzer  is  then  used  to  analyze  this  trace  history  and 
report  on  the  coverage  achieved.  For  each  program  unit,  a  Test  Coverage  Summary  report 
identifies  the  number  of  times  that  unit  was  executed,  a  count  of  the  statements  not  execut¬ 
ed,  and  the  number  of  branch,  basis,  and  complete  paths  that  were  executed.  Further  details 
on  which  statements  were  executed  and  how  many  times  these  were  exercised  are  provided 
by  annotating  a  program  listing.  Similarly,  the  user  can  request  a  report  that  details  those 
paths  that  were  not  exercised.  Reporting  on  the  coverage  accumulated  over  a  series  of  test 
runs  is  achieved  by  requesting  analysis  of  multiple  trace  histories.  Although  this  reports  on 
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the  total  coverage  achieved  with  the  related  test  data,  the  Test  Coverage  Summary  does  not 
distinguish  between  the  coverage  achieved  on  different  runs. 

Additional  TestGen  utilities  are  provided  to  compute  the  cyclomatic  complexity  for 
each  selected  program  unit,  identify  any  unexecutable  paths,  and  provide  a  control  graph 
of  the  code.  A  final  utility,  the  Design  Review  Expert  Assistant,  was  not  examined. 


28.0  QualGen  Overview 

QualGen  is  currently  packaged  as  part  of  the  ADADL  processor.  It  provides  analysis  of 
both  design  and  code  metrics.  QualGen  comes  with  more  than  two  hundred  primitive  met¬ 
rics  divided  into  major  categories  such  as  complexity,  modularity,  documentation,  error 
handling,  system  independence,  and  clean  up.  These  metrics  are  taken  from  the  works  of 
quality  experts  such  as  Boehm,  Halstead,  and  McCabe.  The  Software  Productivity  Consor¬ 
tium  style  guide  was  another  a  source  of  quality  measurement  guidance  used  by  Software 
Systems  Design.  The  user  can  select  which  of  the  primitive  metrics  will  be  reported  on. 

QualGen  results  are  imported  into  Lotus  1-2-3  for  further  analysis  and  reporting.  Using 
Lotus,  the  user  can  create  formulae  that  define  how  primitive  metrics  should  be  combined 
to  yield  higher-level  metrics  such  as  completeness,  reliability,  and  portability.  Lotus  also 
supports  the  preparation  of  graphical  presentation  of  quality  results. 


28.1.4  GrafBrowse  Overview 

Primarily  intended  to  support  reverse  engineering,  GrafBrowse  facilitates  program  un¬ 
derstanding  by  allowing  the  user  to  graphically  view  the  interrelationships  between  Ada  en¬ 
tities.  As  with  the  ADADL  processor,  it  operates  on  both  an  ADADL  design  and  Ada  code. 
Again,  however,  the  IDA  examination  focused  on  the  application  of  GrafBrowse  to  Ada 
code. 

When  invoked,  GrafBrowse  presents  the  program  invocation  tree.  Alternatively  the 
user  can  select  a  declaration  tree  or  called-by  tree.  (The  invocation  and  called-by  trees  can 
be  displayed  in  compact  form,  that  is,  with  lines  that  potentially  cross,  or  in  a  flat  view  with¬ 
out  crossing  lines.)  Where  appropriate,  the  user  can  follow  this  selection  with  another  to 
select  particular  program  units  to  focus  on.  Once  a  picture  is  displayed,  a  pop-up  menu  al¬ 
lows  the  selection  of  a  new  view,  browsing  the  related  source  code,  or  presentation  of  any 
definition  provided  for  the  unit.  Another  menu  allows  some  reformatting  of  the  displayed 
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view,  annotating  each  unit  representation  with  its  formal  parameters,  and  printing  the  view. 
Since  many  of  the  generated  charts  will  be  large,  the  print  option  allows  the  user  to  request 
reducing  a  view  to  fit  on  a  page,  dissecting  and  scaling  a  chart  for  presentation  on  connected 
sheets  of  paper,  or  printing  a  chart  as  it  appears  on  the  screen  using  as  many  sheets  of  paper 
as  necessary. 


28.2  Observations 

Ease  of  use.  The  tools  may  be  invoked  independently  from  the  command  line.  Alterna¬ 
tively,  the  AISLE  user  interface  provides  a  graphical  interface  to  their  use.  In  either  event, 
all  the  tools  are  menu  driven.  An  on-line  manual  is  available  to  provide  on-line  help.  Error 
messages  are  terse  and  only  minimal  checking  of  user  input  is  provided.  No  special  knowl¬ 
edge  is  required  to  use  the  tools.  All  output  reports  are  well-structured  and  provide  easy-to- 
read  information.  A  major  strength  of  this  tool  is  the  clear  identification  of  the  path  condi¬ 
tions  that  guide  the  execution  of  particular  program  paths.  Other  than  setting  default  values 
for  files  names,  the  user  interface  cannot  be  tailored. 

Documentation  and  user  support.  The  documentation  was  helpful  and  included  sev¬ 
eral  useful  examples.  Software  Systems  Design  staff  provided  quick  and  helpful  support. 

Instrumentation  overhead.  The  entire  program  must  be  analyzed  by  the  ADADL  pro¬ 
cessor  before  the  Unit  Test  Strategy  Generator  can  be  used.  Subsequently,  TestGen  func¬ 
tions  can  be  invoked  for  the  files  analyzed  by  the  ADADL  processor.  The  size  of 
instrumented  code  is  minimized  by  allowing  the  user  to  specify  which  modules  in  the  se¬ 
lected  file  should  be  instrumented.  All  selected  modules  are  instrumented  in  the  same  fash¬ 
ion.  Instrumentation  of  the  Ada  Lexical  Analyzer  Generator  gave  a  150%  increase  in  code 
size,  and  an  increase  of  27%  in  the  object  code. 

Ada  restrictions.  The  Unit  Test  Strategy  Generator  can  analyze  any  Ada  code,  but  the 
Test  Coverage  Analyzer  cannot  instrument  tasks.  Source  lines  with  multiple  statements 
may  be  instrumented  incorrectly. 

Problems  encountered.  Execution  of  the  fully  instrumented  Ada  Lexical  Generator 
caused  storage  errors  to  be  raised.  It  is  uncertain  whether  this  problem  was  due  to  the 
amount  of  tracing  data  being  generated  or  to  a  fault  in  the  instrumentation  itself.  Software 
Systems  Design  are  investigating  this  problem. 
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28.3  Planned  Additions 

Software  Systems  Design  is  currently  revising  QualGen.  This  tool  is  being  extracted 
from  the  ADADL  processor  and  being  made  independent  of  Lotus  1-2-3.  It  will  itself  gen¬ 
erate  histograms  and  Kiviat  diagrams  for  presentation  of  metrics  data  and  allow  the  user  to 
combine  metrics  into  composite  quality  measures.  The  new  version  will  also  provide  for 
trend  analysis  of  metrics  data. 

A  new  tool,  called  BugFinder,  will  examine  program  paths  to  look  for  potentially  erro¬ 
neous  conditions,  for  example,  a  path  in  which  the  output  is  not  set.  This  tool  is  expected 
to  become  available  in  April  1993. 

28.4  Sample  Outputs 

Figures  28-1  through  28-29  provide  sample  outputs  from  the  ADADL  processor, 
TestGen,  QualGen,  and  GrafBrowse. 
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nun  non ini  nun iimiiiiinnn  mi 


ADADL  PROCESSOR 

VERSION  5.3.E 
SERIAL  NUMBER:  LOAN— IDA 

AUTHORIZED  USER 
Software  Systems  Design 


(]  [1 

unnnuuuuu  uunnn  imnnn  nn 


PAGE  4 

function  Llfindl  Xtea:  Llstrings;  Which:  Llstyle  )  return  Integer  t 

is 

Find  item  in  symbol  table  -  -  return  index  or  0  if  not  found. 

--  Assumes  symbol  table  is  sorted  in  ascending  order. 

Low,  Midpoint,  High:  Integer ; 
begin 


low  :•  i; 

High  :m  LI tables! se  +  1; 
while  low  /m  High  loop 
|  Midpoint  :«  (High  ♦  Low)  /  2; 

|  if  Item  <  LI symbol table (Midpoint) .Key  then 
j  |  High  : «  Midpoint ; 

j  elsif  Item  -  Llsyaboltable (Midpoint) .Key  then 
I  I  if  Llsyaboltable (Midpoint) .Kind  ■  which  then 

. -return  (  Midpoint  ) ; 

I  I  SlSS 

. return (  0  ) ; 

I  I  saS  it; 

I  Wise  •-  ITEM  >  LLSYMBOLTABLS  (MIDPOIirr)  .KEY 
]  |  Low  : -  Midpoint  ♦  1; 

I  sag  it; 

end  loop; 

- return (  0  );  --  itesi  is  not  in  table 

end  Llfind; 


Figure  28-1.  ADADL  Listing 
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PROGRAM  OMIT  CROSS  REFERBJCE  REPORT 

DEC/REF  LOCATION  OF  DECLARATION  OR  REFERENCE 

PAGE  NO.  LINE  NO.  ENCLOSING  PROGRAM  UNIT 


Advance  <<Procadura  specification) > 

daclaration 

15 

348 

Package  Ll_Tokens 

Buildright  <  <Prooadura  >  > 

daclaration 

19 

303 

Procedure  Readgraa 

code  ref 

31 

394 

Buildsalact  <<Procadura>> 

daclaration 

31 

358 

Procedure  Readgraa 

coda  ref 

31 

395 

Close  <  < Procedure  specification) > 

declaration 

•*  LIBRARY 

ee 

Package  Text_Io 

code  ref 

33 

403 

Procedure  Readgraa 

Llfind  < (Function  -  Returns  Integer) > 

declaration 

4 

83 

Procedure  Ll_Coapile 

code  ref 

13 

315 

Function  Maka_Token 

code  ref 

13 

317 

code  ref 

13 

319 

code  raf 

13 

333 

oode  raf 

13 

334 

code  raf 

13 

337 

code  ref 

39 

563 

Procedure  Parse 

oode  raf 

39 

565 

I.lnain  <  (Procedure)) 

declaration 

17 

369 

Procedure  Ll_Coepile 

oode  ref 

30 

613 

Llnezt token  < (Procedure ) ) 

declaration 

3 

79 

Procedure  Ll_Coapile 

code  ref 

7 

136 

Procedure  Llskiptoken 

oode  ref 

9 

163 

Procedure  Llskipbotk 

code  ref 

39 

534 

Procedure  Synchronise 

code  ref 

39 

566 

Procedure  Parse 

oode  ref 

30 

573 

declaration 

16 

355 

Procedure  LlCoapile 

Figure  28-2.  ADADL  Program  Unit  Cross  Reference  Report 
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OBJECT  CROSS  REFERENCE  REPORT 


PAGE  47 


DEC /REF  LOCATION  OF  DECLARATION  OR  REPERBKZ 

PAGE  NO.  LINE  NO.  ENCLOSING  PROGRAM  UNIT 


Attribute  <<object>> 


declared  as 

->  Llattributa 

dadaration 

12 

196 

Function  Maka_Tokan 

sat 

cod*  r*f 

13 

233 

sat 

cod*  raf 

13 

235 

sat 

coda  raf 

It 

237 

sat 

coda  raf 

14 

239 

sat 

coda  xaf 

14 

241 

usa 

coda  xaf 

14 

243 

Axlcai  <<Objact» 

deolaxad  aa 

•>  Intagar 

daclaratioB 

17 

292 

Procadura  Llaain 

paraaatar 

coda  raf 

21 

367 

Procadura  Raadgxaa 

uaa 

coda  raf 

29 

561 

Procadura  Paraa 

Caaaindax  «In 

[ 

ft 

II 

V 

V 

daclarad  a a 

->  in  Intagar 

dadaration 

16 

266 

Procadura  LI tak auction 

Ch  <<Objaet>> 

daclarad  aa 

«>  Charactar 

dadaration 

18 

298 

Procadura  Raadgraa 

paraaatar 

ooda  raf 

19 

311 

Procadura  Buildright 

uaa 

coda  raf 

19 

312 

paraaatar 

coda  raf 

21 

377 

Procadura  Raadgraa 

uaa 

ooda  raf 

21 

379 

Childoount  <<Objact>> 

daclarad  aa 

->  Intagar 

dadaration 

19 

303 

Procadura  Buildright 

sat 

coda  raf 

19 

307 

sat 

ooda  raf 

19 

314 

uaa 

ooda  raf 

19 

314 

uaa 

ooda  raf 

19 

315 

sat 

ooda  raf 

19 

331 

uaa 

ooda  raf 

19 

321 

uaa 

ooda  raf 

19 

322 

aa t 

ooda  raf 

19 

326 

uaa 

ooda  raf 

19 

326 

uaa 

ooda  raf 

19 

327 

Cr  <<Objact>> 

daclarad  aa 

“>  conatant  Charactar  «■  CR 

dadaration 

**  LIBRARY 

a* 

Paokaga  Aaoii 

uaa 

ooda  raf 

11 

185 

Procadura  a 

Cat  Charactar 


Figure  28*3.  ADADL  Object  Cross  Reference  Report 
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TYPE  CROSS  REFERENCE  REPORT 


DEC/REF  LOCATION  Of  DECLARATION  OR  REFERENCE 

PAGE  NO.  LINE  NO.  BtCLOSING  PROGRAM  UNIT 

Boolean  <<Typ*>> 


declared  a>  ->  (Falae,  True) 

declaration 

»»  LIBRARY 

eft 

Package  Standard 

coda  dac 

1 

60 

Procedure  Ll_Caapile 

coda  dae 

3 

68 

coda  dac 

3 

69 

coda  dac 

11 

178 

Procedure  a 

Get_Character 

aoda  dac 

11 

178 

coda  dao 

15 

248 

Procedure  a 

■pacification  Advance 

code  dec 

15 

248 

coda  dac 

17 

272 

Procedure  Llnain 

Character  <<Type>> 

declared  aa  ->  UNKNOWN 

••• 

declaration 

«•  LIBRARY 

•• 

Package  Standard 

coda  r af 

6 

123 

Procedure  Llprttoken 

coda  dao 

11 

178 

Procedure  a 

Get_Charaoter 

coda  dac 

18 

298 

Procedure  Raadgraa 

rllo_Type  <  <Typa  >  > 

declared  aa  ■»>  Halted  private 

declaration 

*•  LIBRARY 

ft* 

Package  Text_lo 

oode  dae 

18 

299 

Procedure  Readgraa 

Integer  «Type>> 

declared  aa  ->  (Iaplaawntatlondaflned) 

declaration 

••  LIBRARY 

■■ 

Package  Standard 

oode  dac 

2 

35 

Procedure  Ll_Ccapila 

coda  dac 

2 

36 

oode  dec 

3 

52 

coda  dac 

3 

53 

coda  dac 

3 

54 

oode  dec 

3 

56 

oode  dao 

3 

61 

code  dec 

3 

62 

oode  dec 

3 

70 

coda  dac 

3 

71 

coda  dac 

4 

83 

Function  Hfind 

oode  dac 

4 

84 

oode  dao 

12 

195 

Function  MakeJXoken 

oode  dec 

16 

266 

Procedure  Lltakeaotion 

oode  dao 

17 

275 

Procedure  Lima in 

oode  dec 

17 

276 

oode  dao 

17 

280 

Figure  28-4.  ADADL  Type  Cross  Reference  Report 
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DECLAMATION  TREE 


PAGE  NO.  LINE  NO. 


1 

24 

Procedure  Ll_Coaplle 

3 

79 

1 

Procedure  (pacification  Llnexttoken 

4 

83 

1 

Function  Llfind 

5 

106 

1 

Procedure  Llprtatring 

6 

117 

1 

Procedure  Llprttokan 

7 

128 

1 

Procedure  Uakiptokan 

B 

140 

1 

Procedure  Llf kipnode 

9 

153 

1 

Procedure  Llakipboth 

10 

167 

1 

Procedure  Llf * tel 

11 

176 

1 

Procedure  Get_Character 

12 

193 

1 

Function  Make_Token 

13 

198 

1 

|  Function  Cvt_String 

15 

247 

1 

Package  Ll_Token* 

IS 

248 

1 

|  Procedure  apecification  Advance 

15 

252 

1 

Package  body  Ll_Tokena 

16 

255 

1 

Procedure  Llnexttoken 

16 

266 

1 

Procedure  Lltakeaction 

17 

269 

1 

Procedure  1.1  main 

16 

297 

1 

|  Procedure  Readgran 

19 

302 

1 

|  |  Procedure  Buildright 

21 

358 

1 

j  |  Procedure  Buildaelect 

23 

407 

1 

|  Procedure  Parae 

24 

412 

! 

|  |  Procedure  Eraae 

24 

426 

1 

j  |  Procedure  apecification  Teatayncb 

25 

431 

1 

|  |  Procedure  Expend 

26 

436 

1 

j  j  |  Function  Match 

28 

489 

1 

j  j  Procedure  Teatayncb 

26 

490 

1 

j  |  |  Procedure  Synchronise 

PAGE  63 


Figure  28-5.  AOADL  Declaration  Tree 
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PAGE  NO.  LINE  NO. 

MB 

1 

24 

Procedure  Ll_Ccopile 

17 

269 

|  Procedure  Llaain 

16 

297 

1  1 

Procedure  Readgram 

•*  LIBRARY 

•  • 

1  1 

1 

Procedure  specification  Open 

••  LIBRARY 

*• 

1  1 

1 

Procedure  specification  Get 

**  LIBRARY 

*• 

1  1 

1 

Procedure  specification  Sklp_Line 

19 

302 

1  1 

1 

Procedure  Buildright 

••  LIBRARY 

*« 

1  1 

1 

|  Procedure  specification  Get 

**  LIBRARY 

** 

1  ! 

1 

j  procedure  specification  Put 

**  LIBRARY 

*• 

1  1 

I 

j  Function  specification  End_Of_Line 

**  LIBRARY 

*• 

1  1 

1 

j  procedure  specification  Sklp_Llne 

••  LIBRARY 

•  • 

1  1 

1 

j  procedure  specification  Put_Line 

21 

358 

1  1 

1 

Procedure  Builds elect 

■*  LIBRARY 

•  • 

1  1 

1 

|  Procedure  specification  Get 

•«  LIBRARY 

•  • 

1  1 

1 

|  procedure  specification  Skip_Line 

••  LIBRARY 

•  • 

1  1 

1 

Procedure  specification  Close 

23 

407 

1  1 

Procedure  Peree 

i 

63 

1  I 

1 

function  Llfind 

16 

255 

1  1 

1 

Procedure  Llnext  token 

16 

266 

1  1 

1 

Procedure  Lltakeaction 

26 

489 

1  1 

1 

Procedure  Teatsynch 

10 

167 

1  1 

1 

|  Procedure  Llfatal 

•«  LIBRARY 

•  * 

i  1 

1 

|  |  Procedure  specif icetlon  Put 

6 

117 

1  1 

1 

|  |  Procedure  Llprt token 

5 

106 

1  1 

1 

j  j  |  Procedure  Llprtstring 

••  LIBRARY 

«« 

1  1 

1 

1111  Procedure  specification  Put 

••  LIBRARY 

1  I 

1 

|  1  |  Procedure  specification  Put 

••  LIBRARY 

•  • 

1  1 

1 

j  j  Procedure  specification  Put_Line 

26 

490 

1  1 

1 

j  Procedure  Synchronise 

••  LIBRARY 

•  * 

1  1 

1 

j  |  Procedure  specification  Put 

6 

117 

1  1 

1 

j  |  Procedure  Llpxttokan 

5 

106 

1  1 

1 

|  j  |  Procedure  Llprtstring 

••  LIBRARY 

•  • 

1  1 

1 

j  |  |  |  Procedure  specification  Put 

••  LIBRARY 

*• 

1  1 

1 

!  1  j  Procedure  specification  Put 

S 

106 

1  1 

1 

j  |  Procedure  Llprtstring 

**  LIBRARY 

•  • 

1  1 

1 

|  1  |  Procedure  specification  put 

••  LIBRARY 

** 

1  1 

1 

j  |  Procedure  specification  PutjLine 

16 

266 

1  1 

1 

j  |  Procedure  Lltakeaction 

16 

255 

1 

|  |  procedure  Llsexttoken 

25 

431 

!  1 

1 

Procedure  Rspand 

26 

436 

1  1 

1 

|  function  Match 

••  LIBRARY 

«* 

1  1 

1 

j  Procedure  specification  Fut.Liue 

10 

167 

1  1 

1 

j  Procedure  Llfatal 

••  LIBRARY 

*« 

1  1 

1 

|  |  Procedure  specification  Put 

6 

117 

1  1 

1 

j  j  procedure  Llprt token 

5 

106 

1  1 

1 

i  j  |  Procedure  Llprtstring 

••  LIBRARY 

•  * 

I  1 

1 

1111  Procedure  specification  Put 

Figure  28-6.  ADADL  Invocation  Tree 
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WITH  HIERARCHY 


PAGE  67 


PAGE  NO. 

LINE  NO. 

1 

34 

Procedure  Ll_Coapile 

*•  LIBRARY  ** 

|  Package  Ll_Dedarations 

«•  LIBRARY  •* 

|  Instantiated  Package  Integer_Text_Io 

••  LIBRARY  ** 

|  Package  Text_Io 

INTERRUPT  CROSS  REFERENCE  REPORT 

PAGE  68 

DEC/REF  LOCATION  OF  DECLARATION  OR  REFERENCE 

PAGE  NO.  LINE  NO.  ENCLOSING  PROGRAM  UNIT 


NO  INTERRUPTS  TO  REFERENCE 


PAGE  69 

GENERIC  INSTANTIATION  REPORT 


LOCATION  OF  DECLARATION  OR  INSTANTIATION 
PAGE  NO.  LINE  NO.  ENCLOSING/INSTANTIATED  UNIT 


NO  GENERICS  TO  REPORT 


PAGE  70 


Figure  28*7.  ADADL  Additional  Cross  Reference  Reports 
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EXCEPTION  CROSS  REFERENCE  REPORT 


DEC/REF 


LOCATION  OF  DECLARATION  OR  REFERENCE 


PAGE  NO. 


LINE  NO. 


ENCLOSING  PROGRAM  UNIT 


Parsing_Error  << Except ion >> 

*»  No  Dictionary  Definition  Given  ** 


declaration 

2 

26 

Procedure 

Ll__Ca»pile 

raised 

code  ref 

10 

174 

Procedure 

Llfatal 

raised 

code  ref 

19 

334 

Procedure 

Buildright 

raised 

code  ref 

20 

352 

raised 

code  ref 

29 

532 

Procedure 

Synchronize 

PRAGMA  REPORT 


PAGE  71 


LOCATION  OF  REFERENCE 

PAGE  NO.  LINE  NO.  ENCLOSING  PROGRAM  UNIT 
NO  PRAGMAS  USED 


PAGE  72 


PROGRAM  UNIT  RENAMES  REPORT 


LOCATION  OF  RENAMING  DECLARATION 
PAGE  NO.  LINE  NO.  ENCLOSING  PROGRAM  UNIT 


NO  PROGRAM  UNIT  RENAMES  TO  REPORT 


Figure  28*7  continued:  ADADL  Additional  Cross  Reference  Reports 
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PAGE  73 


COMPLEXITY  SUMMARY  REPORT 


Me  Cabe 
COMPLEXITY 
coda  design 


AD  ALL 
COMPLEXITY 
code  design 


Line  no  Program  Unit  Marne 


10  1  13  1  302  Buildright  Procedure 

«*»*  WARMING  i  the  oode  complexity  measure  for  this  module  is  above  the  f 
maximum  level  10 


2 

1 

2 

1 

358 

Buildselect  Procedure 

3 

1 

3 

1 

198 

Cvt_ String  Function 

3 

1 

3 

1 

412 

Erase  Procedure 

7 

1 

11 

1 

431 

Expand  Procedure 

****  WARNING!  the 

code  complexity  measure 

for  this  module  is  above  the  t 

maximum 

level 

10 

3 

1 

3 

1 

178 

Get_Character  Procedure 

1 

1 

1 

1 

24 

Ll_Compile  Procedure 

1 

1 

1 

1 

252 

Ll_Tokens  Package  body 

1 

1 

1 

1 

167 

Llfatal  Procedure 

5 

1 

6 

1 

83 

Llfind  Function 

1 

1 

1 

1 

269 

Llmain  Procedure 

2 

1 

2 

1 

255 

Llnexttoken  Procedure 

3 

1 

3 

1 

106 

Llprtstring  Procedure 

2 

1 

2 

1 

117 

Llprttoken  Procedure 

1 

1 

1 

1 

153 

Llskipboth  Procedure 

1 

1 

1 

1 

140 

Llskipnode  Procedure 

1 

1 

1 

1 

128 

Llskiptoken  Procedure 

1 

1 

1 

1 

266 

Lltakeaction  Procedure 

11 

1 

11 

1 

193 

Make_Token  Function 

*•*«  WARNING:  the 

oode  complexity  measure 

for  this  module  is  above  the  6 

maximum  level 

10 

4 

1 

S 

1 

436 

Match  Function 

11 

1 

13 

1 

407 

Parse  Procedure 

*•**  WARNING:  the 

oode  complexity  measure 

for  this  module  is  above  the  t 

level 

10 

6 

1 

6 

1 

297 

Readgram  Procedure 

10 

1 

16 

1 

490 

Synchronize  Procedure 

***•  WARNING:  the 

oode  complexity  measure 

for  this  module  is  above  the  t 

maximum 

level 

10 

3 

1 

3 

1 

489 

Teatsyncb  Procedure 

Figure  28-8.  ADADL  Complexity  Summary  Report 
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PROGRAM  UNIT  ID 


PROGRAM  UNIT  ID  REPORT 


PAGE  74 


PROGRAM  UNIT  OR  CONFIGURATION  ITEM 


NO  PROGRAM  UNITS  WITH  PROGRAM  UNIT  IDS 


Figure  28-9.  ADADL  Program  Unit  ID  Report 


PAGE  75 


REPORT  ON  OBJECTS  DECLARED  BUT  NOT  USED 


DEC 


LOCATION  OF  DECLARATION 
PAGE  NO.  LINE  NO.  ENCLOSING  PROGRAM  UNIT 


CaMiadti  <<ln  Parameter >> 
declared  as  ->  In  Integer 
declaration 


16 


More  <<In  Parameter >> 

declared  as  ->  in  Boolean  True 

declaration  11 


Tableindex  <<Object>> 

declared  aa  ->  Integer 

declaration  19 


266  Procedure  Lltakeaction 


178  Procedure  t 
Get  Character 


304  Procedure  Buildright 


Figure  28-10.  ADADL  Objects  Declared  but  Not  Used  Report 


REPORT  ON  TTPES  DECLARED  BUT  NOT  USED 


PAGE  76 


DEC 


LOCATION  OF  DECLARATION 
PAGE  NO.  LINE  NO.  ENCLOSING  PROGRAM  UNIT 


NOTHING  TO  REPORT 


Figure  28-11.  ADADL  Types  Declared  But  Not  Used  Report 
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PAGE  77 

REPORT  ON  PROGRAM  UNITS  DECLARED  BUT  NOT  USED 


DEC  LOCATION  OF  DECLARATION 

PAGE  NO.  LINE  NO.  ENCLOSING  PROGRAM  UNIT 


Advance  <<Procedure  specification) > 

declaration  15 

248 

Paokage  Ll_Tokens 

Get_Character  <  <Procedure >  > 

declaration 

11 

178 

Procedure  Ll_Compile 

Ll_Compile 

<<Procedure>> 

declaration 

1 

24 

Ll_Tokens 

<<Package  body>> 

declaration 

declaration 

15 

15 

247 

252 

Procedure  Ll__Conpile 
Procedure  Ll_Coapile 

LlJTokens 

<<Package>> 

declaration 

15 

247 

Procedure  Ll_Compile 

Llskipboth 

<  <Procedure>  > 

declaration 

9 

153 

Procedure  Ll_Ccoapile 

Llskipnode 

<  <Procedure>  > 

declaration 

8 

140 

Procedure  Ll_Ccaapile 

Llskiptoken 

<<Procedure>> 

declaration 

7 

128 

Procedure  Zil_Caapile 

Make_Token 

<<Function>> 

declaration 

12 

193 

Procedure  Ll_Caupile 

Figure  28-12.  ADADL  Program  Units  Declared  But  Not  Used  Report 
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PAGE  102 

REPORT  ON  PROGRAM  UNITS  WITH  HIGH  COMPLEXITY  METRICS 


.INE  NO 

DESIGN  COMPLEXITY  CODE  COMPLEXITY 

PROGRAM  UNIT  NAME 

302 

13 

Buildright  <<  s 

Procedure  >> 

431 

11 

Expand  <  <  t 

Procedure  >  > 

• 

191 

11 

Make_Token  <  <  6 

Function  >> 

407 

13 

Parse  <<  Procedure  t 

>> 

490 

16 

Synchronize  <  <  & 

Procedure  >> 

• 

Figure  28-13.  ADADL  Program  Units  with  High  Complexity  Metrics  Report 

• 

PAGE  103 

• 

ERROR  CROSS  REFERENCE  REPORT 

PAGE  NO.  LINE  NO.  ERROR  MESSAGE 


PAGE  NO.  LINE  NO.  ERROR  MESSAGE 

73  104  *****  WARNING:  Code  complexity  measure  s 

for  a  nodule  ia  above  the  naxlnun  level. 

73  104  *****  WARNING:  Code  complexity  measure  t 

for  a  nodule  is  above  the  naxinun  level. 

73  104  *****  WARNING:  Code  complexity  measure  t 

for  a  nodule  is  above  the  naxinun  level. 

73  104  *****  WARNING:  Code  complexity  measure  s 

for  a  module  is  above  the  naxinun  level. 


73 


104  *****  WARNING:  Code  complexity  measure  t 
for  a  nodule  is  above  the  maximum  level. 


Figure  28-14.  ADADL  Error  Cross  Reference  Report 


PART  il 


TestGen  Family 


*  Testing  all  paths  of  Subprogram  Llfind 


Test  conditions  case  1  of  4  for  subprogram:  Llfind 

Test  conditions  required  for  test  case  1  are: 

90:  Set  (Item  <  Llsymboltable (Midpoint) .Key  )  to  False 
92:  Set  (Item  -  Llsymboltable (Midpoint) . Key  )  to  False 

Statements  to  be  executed  during  test  case  1  are: 

83:  Procedure  Llfind  is 

85:  Begin 

86:  Low  :-  1; 

87:  High  :-  Lltablesize  +  1; 

88:  While  Low  /-  High  loop 
89:  Midpoint  : -  (High  +  Low)  /  2; 

90:  If  Item  <  Llsymboltable (Midpoint) .Key  then 
•**  Condition  is  False 

92:  Elsif  Item  -  Llsymboltable (Midpoint) .Key 
**«  Condition  is  False 

98:  Else 

99:  Low  : “  Midpoint  +  1; 

100:  End  if  —  for  90 
101 :  End  Loop 

**•  Exit  loop  at  88  when  (Low  /•  High  )  is  falsa. 

102:  Return  (  0  ); 

103 :  End 

Test  conditions  case  2  of  4  for  subprogram:  Llfind 

Test  conditions  required  for  test  oase  2  are: 

90:  Set  (Item  <  Llsymboltable (Midpoint) .Key  )  to  False 
92:  Set  (Item  -  Llsymboltable (Midpoint) .Key  )  to  True 
93:  Set  ( Llsymboltable ( Midpoint) .Kind  “  Which  )  to  False 

Statements  to  be  executed  during  test  case  2  are: 

83:  Procedure  Llfind  is 

85:  Begin 

86:  Low  If 

87:  High  Lltablesize  +  1; 

88:  While  Low  /-  High  loop 
89:  Midpoint  :-  (High  +  Low)  /  2; 

90:  If  Item  <  Llsymboltable (Midpoint) .Key  then 
•••  Condition  is  False 

92:  Elsif  Item  -  Llsymboltable (Midpoint) .Key 
***  Condition  is  True 

93:  If  Llsymboltable (Midpoint) .Kind  “  Which  then 
Condition  is  False 

95:  Else 

Figure  28*15.  TestGen  Test  Conditions  for  Path  Testing  of  LLFIND 
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96 :  Return  (  0  )  ; 
103 s  End 


Test  conditions  case  3  of  4  for  subprogram :  Llfind 

Test  conditions  required  for  test  case  3  are: 

90:  Set  (Item  <  Llsymboltable(Midpoint) . Key  )  to  False 
92:  Set  (Item  •  Llsyaboltabla (Midpoint) .Key  )  to  True 
93:  Set  (Llsymboltable(Midpoint) .Kind  -  which  )  to  True 


Statements  to  be  executed  during  test  case  3  are: 

83:  Procedure  Llfind  is 

85:  Begin 

86:  Low  1; 

87:  High  : -  Lltablesize  +  1; 

88:  While  Low  /-  High  loop 
89:  Midpoint  :«  (High  +  Low)  /  2; 

90:  If  Item  <  Llsymboltable( Midpoint) .Key  then 
Condition  is  False 

92:  Elsif  Item  »  Llsymboltable (Midpoint ). Key 
***  Condition  is  True 

93:  If  Llsymboltable (Midpoint) .Kind  -  Which  -then 
»•*  Condition  is  True 
94:  Return  (  Midpoint  ); 

103 :  End 


Test  conditions  case  4  of  4  for  subprogram:  Llfind 


Test  conditions  required  for  test  case  4  are: 

90:  Set  (Item  <  Llsymboltable (Midpoint) .Key  )  to  True 


Statements  to  be  executed  during  test  case  4  are: 


83:  Procedure  Llfind  is 

85 :  Begin 

86:  LOW  :-  1; 

87:  High  Lltablesize  +  1; 

88:  While  Low  /-  High  loop 
89:  Midpoint  :-  (High  +  Low)  /  2; 

90:  If  Item  <  Llsymboltable (Midpoint) .Key  then 
***  Condition  is  True 
91:  High  :■  Midpoint; 

100:  End  if  —  for  90 
101 :  End  Loop 

***  Exit  loop  at  88  when  (Low  /-  High  )  is  false. 
102:  Return  (  0  )/ 

103:  End 


Figure  26-15  continued:  TestGen  Test  Conditions  for  Path  Testing  of  LLFIND 
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Test  Case  Effort  Report 

page  1  of  2 

Number  of  Test  Cases  Required  for 

Module  Name  Basis  Branch  Full  Path 

Testing  Testing  Testing 


i  Ll_Compile 
!  Llfind 
!  Llprtstxing 
|  Llprttoken 
|  Llskiptoken 
i  Llakipnode 
(  Llskipboth 
I  Llfatal 
I  Get_Character 
I  Make_Token 
I  Cvt_String 
[  Lin ext token 
I  Llmain 


Test  Case  Effort  Report 

page  2  of  2 

Number  of  Test  Cases  Required  fox 

Module  Name  Basis  Branch  Full  Path 

Testing  Testing  Testing 


|  Readgrsm 

1  1 

|  s 

1  2 

1  Buildright 

1  9 

|  7 

|  22 

I  Builds elect 

I  1 

|  1 

I  1 

I  Parse 

1  io 

1  8 

!  32 

1  Erase 

1  2 

1  2 

1  2 

1  Expand 

1  6 

|  4 

1  10 

j  Match 

|  3 

|  3 

1  3 

I  Testsynch 

I  2 

1  2 

|  2 

1  Synchronise 

1  o 

|  4 

|  10 

Figure  28*16.  TestGen  Test  Case  Effort  Report 
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Unreachable  Statement  Report  for  module:  Ll_Compile 
All  statements  can  be  reached. 

Unreachable  Statement  Report  for  module:  Llflnd 
97:  End  if  --  for  93 

There  were  1  statements  that  could  not  be  reached 

Unreachable  Statement  Report  for  module:  Llprtstring 
All  statements  can  be  reached. 

Unreachable  Statement  Report  for  module:  Llprttoken 
All  statements  can  be  reached. 

Unreachable  Statement  Report  for  module:  Llskiptoken 
All  statements  can  be  reached. 

Unreachable  Statement  Report  for  module:  Llskipnode 
All  statements  can  be  reached. 

Unreachable  Statement  Report  for  module:  Llskipboth 
All  statements  can  be  reached. 

Unreachable  Statement  Report  for  module:  Llfatal 
All  statements  can  be  reached. 

Unreachable  Statement  Report  for  module:  Get_Charaater 
All  statements  can  be  reached. 

Unreachable  Statement  Report  for  module:  Kake_Token 
All  statements  can  be  reached. 

Unreachable  Statement  Report  for  module:  Cvt_Strlng 
All  statements  can  be  reached. 

Unreachable  Statement  Report  for  module:  Llnexttoken 
All  statements  can  be  reached. 

Unreachable  Statement  Report  for  module:  Umain 
All  statements  can  be  reached. 

Unreachable  Statement  Report  for  module:  Readgram 
All  statements  can  be  reached. 

Unreachable  Statement  Report  for  module:  Bulldright 
All  statements  can  be  reached. 

Unreachable  Statement  Report  for  module:  Buildselect 
All  statements  can  be  reached. 

Unreaohable  Statement  Report  for  module:  Parse 
All  statements  can  be  reached. 


Figure  28-17.  TestGen  Unreachable  Statement  Report  for  LL.COMPILE 
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McCabe  Cyclonatic  Complexity  Report 


page  1  of  2 

Module  Name  Design  Code 


|  UL_Ccanpile 

|  1 

|  1 

|  LI find 

j  1 

j  5 

|  Llprtstring 

j  1 

|  3 

|  Llprttoken 

|  1 

|  2 

|  Llskiptoken 

j  1 

j  1 

|  Llskipnode 

j  1 

j  1 

j  Llskipboth 

|  1 

|  1 

|  LI fatal 

|  1 

j  1 

j  Get  Character 

i  1 

1  3 

|  Make_Token 

j  1 

j  11 

j  Cvt_String 

|  1 

j  3 

|  Llnexttoken 

I  1 

1  2 

j  I.lmain 

|  1 

|  1 

|  Readgram 

j  1 

1  6 

|  Buildright 

j  1 

j  10 

McCabe  Cyclonatic  Complexity  Report 

page  2  of  2 

Module  Name 

Design 

Code 

|  Buildselect 

1  1 

i  2 

1 

j  Parse 

I  1 

1  11 

1 

|  Erase 

j  1 

j  3 

1 

|  Expand 

j  1 

j  7 

1 

j  Match 

j  1 

i  4 

1 

|  Testsynch 

I  1 

1  3 

1 

|  Synchronize 

j  1 

j  9 

1 

ee 

Figure  28-18.  TestGen  McCabe  Complexity  Report  for  LL_COMPILE 
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Test  Coverage  Surma ry 


Page  1  of  3 


Module 

Name 

Calls 

Stats 

Not 

Done 

Complete 

Path 

Branch 

Path 

Basis 

Path 

Ll_Compile 

|  1 

0  1 

1/1 

(100%) | 

1/1 

(100%)| 

1/1 

(ioo%)| 

Llfind 

j  19$ 

4  | 

3/4 

(  75%) | 

3/4 

(  75%)| 

3/4 

( 

75%)  | 

Llprt string 

!  o 

10  j 

0/2 

(  0%)| 

0/2 

(  0%)| 

0/1 

( 

o%)| 

Llprttoken 

1  0 

10  j 

0/2 

(  0%)  | 

0/2 

(  0%)| 

0/2 

( 

0%)| 

Llskiptoken 

|  0 

11  1 

0/1 

(  0%)| 

0/1 

(  0%)| 

0/1 

( 

0%)| 

Llskipnode 

|  0 

12  j 

0/1 

{  0%)  | 

0/1 

(  0%)| 

0/1 

( 

0%)l 

Llskipboth 

|  0 

13  I 

0/1 

(  0%)| 

0/1 

(  0%)| 

0/1 

( 

0%)| 

Ll fatal 

1  0 

10  j 

0/1 

(  0%)| 

0/1 

(  0%)| 

0/1 

( 

0%)| 

GetjCharacter 

|  865 

1  j 

3/3 

(100%) | 

3/3 

(100%)| 

3/3 

(ioo%)| 

Make_Token 

1  133 

9  1 

5/35 

(  14%) I 

5/7 

(  71%) | 

1/11 

( 

9%)l 

- - - 


Test  Coverage  Summary 

Page  2  of  3 

Module  Calls  Stmts  Complete  Branch 

Name  Not  Path  Path 

Done 


Basis 

Path 


f 

1 

CvtString 

1  133  | 

1 

1 

2/2 

(100%) | 

2/2 

(100%)  | 

2/2 

(100%) | 

1 

Llnezttoken 

j  134  | 

1 

1 

2/2 

(100%)| 

2/2 

(ioo%)| 

2/2 

(100%) | 

1 

Llaain 

I  1  | 

1 

1 

1/1 

(100%) | 

1/1 

(100%)  | 

1/1 

(100%) | 

1 

Keadgraa 

1  1  1 

1 

1 

1/2 

(  50%) | 

2/2 

(ioo%)| 

1/2 

(  50%) | 

1 

Buildright 

|  64  j 

7 

1 

8/22 

(  36%) | 

5/7 

(  71%) 1 

3/9 

(  33%) | 

1 

Builds elect 

1  64  j 

1 

I 

1/1 

(100%) | 

1/1 

(100%)  1 

1/1 

(100%) | 

Parse 

|  1  j 

12 

1 

5/32 

(  16%) I 

2/8 

(  25%)  | 

1/10 

(  10%) | 

1 

Erase 

I  398  j 

1 

1 

2/2 

(100%) | 

2/2 

(100%) 1 

2/2 

(100%) | 

1 

Expand 

1  254  | 

5 

1 

5/10 

(  50%) | 

2/4 

(  50%)  1 

2/6 

(  33%) | 

1 

Match 

j  254  j 

4 

1 

1/3 

(  33%) | 

1/3 

(  33%)  | 

1/3 

(  33%) | 

Test  Coverage  Summary 

Page  3  of  3 

Module  Calls  Stats  Complete  Branch 

Name  Not  Path  Path 

Done 


Basis 

Path 


|  Teatsynch 

1  o  | 

12 

I 

0/2 

( 

0%)[ 

0/2 

(  0%)  | 

0/2 

- - 1 

(  0%)  1 

|  Synchronize 

1  o  | 

44 

1 

0/10 

( 

0%)| 

0/4 

(  0%) 

0/6 

(  0%)  | 

j  Ll_Tokens 

I  0  j 

3 

1 

o/i 

( 

0%)i 

0/1 

(  o%)| 

0/1 

(  0%)  1 

j  Current_Symbol 

|  133  j 

1 

1 

1/1 

(ioo%)| 

1/1 

(ioo%)| 

1/1 

(100%) 1 

|  Advance 

|  134  j 

7 

1 

5/8 

( 

62%)  | 

6/8 

(  75%) | 

5/8 

(  52%) | 

|  SoanJPattern 

|  257  j 

129 

1 

16/585 

(  3%) 

12/40 

(  30%) | 

1/50 

(  2%) 

j  Char_Advance 

j  780  J 

3 

1 

1/3 

< 

33%)  | 

1/3 

(  33%)  | 

1/3 

(  33%)  | 

j  Look_Ahead 

1  39  j 

2 

1 

1/3 

( 

33%)  | 

1/3 

(  33%) | 

1/3 

(  33%)  | 

j  Lltak section 

j  230  j 

77 

1 

32/68 

( 

47%  )| 

32/68 

(  47%) | 

32/68 

(  47%) | 

Figure  28*19.  TestGen  Test  Coverage  Summary  using  testl.lex 
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TestGen  Sub-Program  Invocation  Count  Report 


Module  Name 

Invocations 

Ll_Compile 

1 

1 

LI find 

1 

198 

Llprtstring 

1 

0 

Llprt token 

1 

0 

Llskiptoken 

! 

0 

Llskipnode 

1 

0 

Llskipboth 

I 

0 

LI fatal 

1 

0 

Get  Character 

1 

865 

Make  Token 

1 

133 

Cvt_String 

1 

133 

Llnezttoken 

1 

134 

Lima in 

1 

1 

Readgram 

1 

1 

Buildright 

1 

64 

Buildaelect 

1 

64 

Parse 

1 

1 

Erase 

1 

398 

Expand 

1 

354 

Match 

t 

354 

Testsynch 

i 

0 

Synchronize 

1 

0 

LlJTokens 

1 

0 

Cur rent_ Symbol 

1 

133 

Advance 

1 

134 

Soan_Pattem 

1 

357 

CharAdvance 

1 

780 

Look_Ahead 

1 

39 

Lltakeaotion 

1 

330 

Figure  28-20.  TestGen  Sub-Program  Invocation  Count  Report  using  testl.lex 
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Statement 


134: 

134: 

134: 

134: 

257: 

257: 

1: 

1: 

1: 

45: 

9: 

9: 

9: 

78: 

64: 

64: 

64: 

60: 

60: 

60: 

0: 

0: 

0: 

0: 

0: 

0: 

123: 

123: 

257: 


Execution  Report  for  Module:  Advance 


procedure  ADVANCE (EOS:  out  BOOLEAN; 

NEXT:  out  LLTOKEN; 

MORE:  In  BOOLEAN  TRUE)  la 
begin 

EOS  FALSE; 
loop 

SCAN_PATTERN; 
case  CUR_PATTERN  la 
when  END_OF_INP0T  -> 

EOS  TRUE; 
return; 

when  END_OF_LINE  ->  null; 
when  Charaoter_Llteral  •> 

NEXT  MAKE_T0KEN (  CHAR,  CURRENT_STMBOL,  CUR_LINE_NUM); 
return; 

when  Comment  |  White_Space  ->  null; 

when  Delimiter  |  Number  |  Special_Symbol  -> 

NEXT  MAKE_TOKEN(  LIT,  CURRENT_STMBOL,  CUR_LINE_NUM) ; 
return; 

when  Identifier  * > 

NEXT  NAKE_TOKEN (  I DENT,  CURRENT_STMBOL,  CUR_L1NE_NUM) / 
return; 

when  String_Llteral  -> 

NEXT  NAK£_TOKEN(  STR,  CURRENT_STMBOL,  CURLINENUM) ; 
return; 

when  others  “> 

NEXT  :«■  MAKE_TOKEN(  LIT,  CURRENT_SYMBOL,  CUR_LINE_NUM) ; 
return; 
end  case; 
end  loop; 
end  ADVANCE; 


Figure  28-21.  TestGen  Statement  Execution  Report  using  test1.lex  for  ADVANCE 
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Branch  Path  Coverage  Analysis  for  Module:  Advance 


There  were  2  paths  not  tested: 


*  Testing  all  statements  of  Subprogram :  Advance 


Test  conditions  case  1  of  2  for  subprogram:  Advance 

Test  conditions  required  for  test  case  1  are: 

51:  Set  (Cur_Pattern  )  to  String_Literal 


Statements  to  be  executed  during  test  case  1  are: 


46: 

47: 

48: 

49: 

50: 

51: 


66: 

67: 

68: 

74: 


Procedure  Advance  is 
Begin 

Eos  :“  False/ 

Loop 

Scan_Pattern ; 

Case  Cur_Pattern  is 

***  Case  variable  is  String_Literal 
When  String_Literal  •> 

Next  :•  Make_Token(  Str,  Current_Symbol ,  Cur_Line_Num) ; 
Return  ; 

End 


Test  conditions  case  2  of  2  for  subprogram:  Advance 

Test  conditions  required  for  test  case  2  are: 

51:  Set  (Cur_Pattern  )  to  others 


Statements  to  be  exeauted  during  test  case  2  are: 

Procedure  Advance  is 
Begin 

Eos  i«  False; 

Loop 

Scan_Pattern; 

Case  Cur_Pattern  is 

*•*  Case  variable  is  others 
When  others  ■> 

Next  : -  Make_Token(  Lit,  Current_Symbol ,  Cur_Line_Num) ; 
Return  / 

End 

2  of  8  paths  were  not  tested. 

This  module  is  75  percent  tested. 


46: 

47: 

48: 

49: 

50: 

51: 

69: 

70: 

71: 

74: 


Figure  28-22.  TestGen  Branch  Path  Coverage  Analysis  using  testl.lex  for  ADVANCE 
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Structured  Testing  Path  Coverage  Analysis  for  Module i  Advance 
There  were  3  paths  not  tested  : 


*  Testing  a  Structured  path  selection  of  Subprogram  Advance 


Test  conditions  case  1  of  3  for  subprogram:  Advance 


Test  conditions  required  for  test  case  1  are: 
SI:  Set  (Cur_Pattern  )  to  End_Of_Input 


Statements  to  be  executed  during  test  case  1  are: 

46:  Procedure  Advance  is 

47 :  Begin 

48:  Eos  :•  False; 

49:  Loop 

50:  Scan_Pattern; 

51:  Case  Cur_Pattern  is 

***  Case  variable  is  End_Of_Input 
52>  When  End_Of_Input  “> 

53:  Eos  :-  True; 

54 :  Return  ; 

74:  End 


Test  conditions  case  2  of  3  for  subprogram:  Advance 


Test  conditions  required  for  test  case  2  are: 
51:  Set  (Cur_Pattern  )  to  String_Literal 


Statements  to  be  executed  during  test  case  2  are: 

46:  Procedure  Advance  is 

47 :  Begin 

48:  Eos  i"  False; 

49:  Loop 

5  0 :  Scan_P attar n ; 

51:  Case  Cur_Pattern  is 

Case  variable  is  String_Literal 
66:  When  String_Literal  -> 

67:  Next  NaJce_Token(  Str,  Current_Symbol ,  Cur_Line_Num) ; 

68:  Return  ; 

74:  End 


Figure  28-23.  TestGen  Structured  Testlnj^Path  Coverage  Analysis  using  testl.lex  for  AD- 
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Test  conditions  required  for  test  case  3  are: 

51:  Set  ( Cur_Pattern  )  to  others 

Stateaents  to  be  executed  during  test  case  3  are: 

46:  Procedure  Advance  is 

47 :  Begin 

48:  Eos  :•  False; 

49:  Loop 

50 i  Scan_Pattern; 

51i  Case  Cur_Pattern  is 

•««  case  variable  is  others 
69:  When  others  -> 

70:  Next  :-  Make_Token (  Lit,  Current_Syabol ,  Cur_Line_Nua) ; 

71:  Return  ; 

74:  End 

3  of  8  paths  were  not  tested. 

This  module  is  62  percent  tested. 


Figure  28*23  continued :  TestGen  Structured  Testing  Path  Coverage  Analysis  using  testl  .lex 

tor  ADVANCE 
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Figure  28*24.  TestGen  Test  Coverage  Summary  using  testl.lex  &  sample.lex 
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Figure  28-25.  QualGen  Report  Excerpt 
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Figure  28-25  continued:  QualGen  Report  Excerpt 
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Figure  28-25  continued:  QualGen  Report  Excerpt 
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Figure  28*28.  Graf  Browse  Flat  Callby  Tree  of  LLFIND 


Figure  28*29.  Grafbrowse  Browsing  LLFINO 
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GLOSSARY 

The  reader  is  assumed  to  be  familiar  with  general  software-related  terms  and,  therefore, 
this  glossary  focuses  on  testing  and  evaluation  terms.  The  reader  is  referred  to  the  IEEE 
Standard  Glossary  of  Software  Engineering  Terminology  [IEEE  1990]  for  definitions  of 
additional  terms. 

Acceptance  Testing.  Formal  testing  conducted  to  determine  whether  or  not  a  system  sat¬ 
isfies  its  acceptance  criteria  and  to  enable  a  customer  to  determine  whether  or  not  to  accept 
the  system. 

Assertion.  A  logical  expression  specifying  a  program  state  that  must  exist  or  a  set  of  con¬ 
ditions  that  program  variables  must  satisfy  at  a  particular  point  during  program  execution; 
for  example,  “A  is  positive  and  greater  than  B." 

Assertion  Testing.  A  technique  which  inserts  assertions  about  a  program  state  or  the  rela¬ 
tionship  between  program  variables  into  the  program  code.  The  truth  of  the  assertions  is 
determined  as  the  program  executes. 

Audit.  (1)  An  independent  review  for  the  purpose  of  assessing  compliance  with  software 
requirements,  specifications,  baselines,  standards,  procedures,  instructions,  codes,  and  con¬ 
tractual  and  licensing  agreements.  (2)  An  activity  to  determine  through  investigation  the 
adequacy  of,  and  adherence  to,  established  procedures,  instructions,  specifications,  codes, 
and  standards  or  other  applicable  contractual  and  licensing  requirements,  and  the  effective¬ 
ness  of  the  implementation. 

Auditing.  Checking  for  conformance  of  code  to  prescribed  programing  standards  and  prac¬ 
tices. 

Basic  Execution  Time  Model.  A  software  reliability  model  in  which  the  failure  process  is 
assumed  to  be  a  nonhomogeneous  Poisson  process  with  linearly  decreasing  failure  intensi¬ 
ty- 

Basis  Paths.  Program  paths  that  have  no  iteration. 
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Block.  ( 1 )  In  problem-oriented  languages,  a  computer  program  subdivision  that  serves  to 
group  related  statements,  delimit  routines,  specify  storage  allocation,  delineate  the  applica¬ 
bility  of  labels,  or  segment  paths  of  the  computer  program  for  other  purposes.  (2)  A  group 
of  contiguous  storage  locations,  computer  program  statements,  records,  words,  character¬ 
istics,  or  bits  that  are  treated  as  a  unit. 

Bottom-up  Testing.  A  systematic  testing  strategy  that  seeks  to  test  those  modules  at  the 
bottom  of  the  invocation  structure  first.  These  modules  are  tested  independently  using  test 
drivers  to  invoke  them,  then  modules  at  the  next  higher  level  that  call  these  modules  are 
tested,  and  so  on. 

Boundary  Value  Analysis.  A  test  data  selection  technique  in  which  test  data  are  chosen  to 
lie  along  boundaries  of  the  input  domain  (or  output  range)  classes,  data  structures,  proce¬ 
dure  parameters,  etc.  Choices  often  include  maximum,  minimum,  and  trivial  values  or  pa¬ 
rameters. 

Branch  Testing.  Testing  designed  to  execute  each  outcome  of  each  decision  point  in  a 
computer  program. 

Calendar  Time.  Chronological  time,  including  time  during  which  a  computer  may  not  be 
running. 

Call  Graph.  A  diagram  that  identifies  the  modules  in  a  system  or  computer  program  and 
shows  which  modules  call  one  another. 

Cause-Effect  Graphing.  A  test  data  selection  technique  where  the  input  and  output  do¬ 
mains  are  partitioned  into  classes  and  analysis  is  performed  to  determine  what  effects  are 
caused  by  what  inputs.  A  minimum  set  of  inputs  is  chosen  that  will  cover  the  entire  effect 
set. 

Change  Request.  A  document  used  to  propose,  transmit,  and  record  changes  to  a  specifi¬ 
cation. 

Clock  Time.  Elapsed  time  from  the  start  to  the  end  of  program  execution,  including  wait 
time,  on  a  running  computer. 

Code  Auditor.  An  automated  tool  which  checks  for  conformance  to  prescribed  program¬ 
ming  standards  and  practices. 
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Code  Inspection.  See  Inspection. 

Code  Review.  A  meeting  at  which  software  code  is  presented  to  project  personnel,  manag¬ 
ers,  users,  customers,  or  other  interested  parties  for  comment  or  approval. 

Comparator.  A  software  tool  which  compares  two  computer  programs,  files,  or  sets  of 
data  to  identify  commonalities  or  differences.  Typical  objects  of  comparison  are  similar 
versions  of  source  code,  object  code,  data  base  files,  or  test  results. 

Complexity.  The  degree  of  complication  of  a  system  or  system  component,  determined  by 
such  factors  as  the  number  and  intricacy  of  interfaces  and  conditional  branches,  the  degree 
of  nesting,  the  types  of  data  structures,  and  other  system  characteristics. 

Component.  One  of  the  parts  of  a  system.  A  component  may  be  hardware  or  software  and 
may  be  subdivided  into  other  components. 

Concurrent  Processes.  Processes  that  may  execute  in  parallel  on  multiple  processors  or 
asynchronously  on  a  single  processor.  Concurrent  processes  may  interact  with  each  other, 
and  one  process  may  suspend  execution  pending  receipt  of  information  from  another  pro¬ 
cess  or  the  occurrence  of  an  external  event. 

Control  Flow.  The  sequence  in  which  operations  are  performed  during  the  execution  of  a 
computer  program. 

Control  Flow  Knot.  A  control  flow  knot  occurs  when  two  control  flow  jumps  cross.  It  is 
defined  as:  A  jump  (P,  Q)  and  another  jump  (A,  B)  give  rise  to  a  knot  if  ( 1 )  P  lies  within  (A, 
B)  and  Q  lies  outside,  or  (2)  Q  lies  within  (A,  B)  and  P  lies  outside.  Variations  on  basic  knots 
include  down-down  knots,  up-down  knots,  and  up-up  knots. 

Correctness.  See  Program  Correctness. 

Coverage  Analyzer.  A  software  tool  which  determines  and  assesses  measures  associated 
with  the  invocation  of  program  structural  elements  to  determine  the  adequacy  of  a  test  run. 

Coverage  Measure.  In  general,  a  measure  of  the  testing  coverage  achieved  as  a  result  of  a 
test,  often  expressed  as  a  percentage  of  the  number  of  statements,  branches,  or  paths  that 
were  traversed. 

Cross-Referencer.  (1)  A  computer  program  that  provides  cross-reference  information  on 
system  components.  For  example,  programs  can  be  cross-referenced  with  other  programs. 
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macros,  and  parameter  names.  This  capability  is  useful  in  assessing  the  impact  of  changes 
to  one  area  or  another.  (2)  A  utility  program  which  provides  cross-reference  data  concern¬ 
ing  a  program  written  in  a  higher  level  language.  These  utility  programs  analyze  a  source 
program  and  provide  as  output  such  data  as  follows:  1.  Statement  label  cross-index,  2.  Data 
name  cross-index,  3.  Literal  usage  cross-index,  4.  Inter-subroutine  call  cross-index,  5.  Sta¬ 
tistical  counts  of  statement  types. 

Cyclomatic  Complexity.  A  measure  of  program  complexity  derived  from  the  control 
graph  of  a  program.  The  cyclomatic  complexity  of  a  program  is  equivalent  to  the  number 
of  decision  statements  plus  1. 

Data  Flow.  The  sequence  in  which  data  transfer,  use,  and  transformation  are  performed 
during  the  execution  of  a  computer  program. 

Data  Flow  Analysis.  Consists  of  the  graphical  analysis  of  collections  of  (sequential)  data 
definitions  and  reference  patterns  to  determine  constraints  that  can  be  placed  on  data  values 
at  various  points  of  executing  the  source  program. 

Data  Flow  Anomaly.  A  sequence  of  the  events  reference  (r),  definition  (d),  and  use  (u)  of 
variables  in  a  program  that  is  either  erroneous  in  itself  or  often  symptomatic  of  an  error. 

Data  Flow  Testing.  A  testing  technique  which  provides  a  set  of  successively  more  strin¬ 
gent  path  selection  criteria  that  guide  the  selection  of  test  data  to  examine  the  relationships 
between  variable  definitions  and  variable  uses. 

Debugger.  A  software  tool  intended  to  assist  the  user  in  software  fault  localization  and,  po¬ 
tentially,  fault  correction. 

Debugging.  The  process  of  correcting  syntactic  and  logical  faults  detected  during  testing. 
Debugging  shares  with  testing  certain  techniques  and  strategies,  but  differs  in  its  usual  ad 
hoc  application  and  local  scope. 

Decision -To- Decision  Path.  A  sequence  of  nodes  on  the  control  graph  that  starts  at  the 
program  entry  point  or  at  a  decision  node,  terminates  with  the  program  exit  or  a  decision 
node,  and  has  no  decision  node  in  between. 

Directed  Graph.  Consists  of  a  set  of  nodes  interconnected  with  oriented  arcs.  An  arbitrary 
directed  graph  (digraph)  may  have  many  entry  nodes  and  many  exit  nodes.  A  program  di¬ 
graph  has  only  one  entry  and  one  exit. 
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Driver.  A  software  module  that  invokes  and,  perhaps,  controls  and  monitors  the  execution 
of  one  or  more  other  software  modules. 

Dynamic  Analysis.  The  process  of  evaluating  a  system  or  component  based  on  its  behavior 
during  execution. 

Emulator.  A  device,  computer  program,  or  system  that  accepts  the  same  inputs  and  pro¬ 
duces  the  same  outputs  as  a  given  system. 

Entry  Point.  A  point  in  a  software  module  at  which  execution  of  the  module  can  begin. 

Equivalence  Class  Partitioning.  A  test  data  selection  technique  based  on  consideration  of 
partitioning  the  input  domain  of  a  program  into  a  finite  number  of  equivalence  classes  such 
that  ( 1 )  a  test  of  a  representative  value  of  each  class  is  equivalent  to  a  test  of  any  other  value 
and  (2)  each  test  case  should  invoke  as  many  different  input  conditions  as  possible  in  order 
to  minimize  the  total  number  of  test  cases  necessary. 

Error.  (1)  A  discrepancy  between  a  computed,  observed,  or  measured  value  or  condition 
and  the  true,  specified,  or  theoretically  correct  value  or  condition.  (2)  A  mental  mistake 
made  by  a  programmer  which  may  result  in  a  program  fault. 

Error  Guessing.  A  test  data  selection  technique.  The  selection  criteria  is  to  pick  values  that 
seem  likely  to  cause  failures. 

Error  Seeding.  The  process  of  intentionally  adding  known  faults  to  those  already  in  a  com¬ 
puter  program  for  the  purpose  of  monitoring  the  rate  of  detection  and  removal,  and  estimat¬ 
ing  the  number  of  faults  remaining  in  the  program. 

Essential  Knots.  A  measure  of  unstructuredness  based  on  control  flow  knots. 

Essential  Paths.  Program  paths  that  must  be  executed  to  achieve  100%  coverage. 

Exception.  An  event  that  causes  suspension  of  normal  program  execution. 

Executable  Specification.  A  specification  which  is  given  in  a  sufficiently  formal  notation 
to  allow  its  execution  by  a  computer. 

Executable  Statement.  A  statement  in  a  module  which  is  executable  in  the  sense  that  it 
produces  object  code  instructions. 
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Execution  Time.  (1)  The  amount  of  actual  or  central  processor  time  used  in  executing  a 
program.  (2)  The  period  of  time  during  which  a  program  is  executing. 

Failure.  The  inability  of  a  system  or  system  component  to  perform  a  required  function 
within  specified  limits.  A  failure  may  be  produced  when  a  fault  is  encountered. 

Failure  Intensity.  Failures  per  unit  of  time,  the  derivative  with  respect  to  time  of  the  mean 
value  function  of  failures. 

Failure  Intensity  Decay  Parameter.  In  the  logarithmic  Poisson  execution  time  model,  the 
parameter  that  represents  the  rate  of  exponential  decay  of  the  failure  intensity  as  a  function 
of  mean  failures  experienced. 

Failure  Severity.  Classification  of  a  failure  by  its  operational  impact. 

Fault.  A  manifestation  of  an  error  in  software.  A  fault,  if  encountered,  may  cause  a  failure. 

Fault  Tree  Analysis.  A  form  of  safety  analysis  that  assesses  hardware  safety  to  provide 
failure  statistics  and  sensitivity  analyses  which  indicate  the  possible  effect  of  critical  fail¬ 
ures. 

Flowchart.  A  control  flow  diagram  in  which  suitably  annotated  geometrical  figures  are 
used  to  represent  operations,  data,  or  equipment,  and  arrows  are  used  to  indicate  the  se¬ 
quential  flow  from  one  to  another. 

Formal  Specification.  In  proof  of  correctness,  a  description  in  a  formal  language  of  the  ex¬ 
ternally  visible  behavior  of  a  system  or  system  component  Generally,  a  specification  writ¬ 
ten  and  approved  in  accordance  with  established  standards. 

Formal  Verification.  See  Verification. 

Function.  (1)  A  specific  purpose  of  an  entity  or  its  characteristic  action.  (2)  A  subprogram 
that  is  invoked  during  the  evaluation  of  an  expression  in  which  its  name  appears  and  that 
returns  a  value  to  the  point  of  invocation. 

Functional  Specification.  A  set  of  behavioral  and  performance  requirements  which,  in  ag¬ 
gregate,  determine  the  functional  properties  of  a  software  system. 

Function  Points.  Function  points  measure  software  by  quantifying  the  functionality  pro¬ 
vided  external  to  itself.  It  is  based  primarily  on  logical  design. 
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Functional  Testing.  Testing  that  ignores  the  internal  mechanism  of  a  system  or  component 
and  focuses  solely  on  the  outputs  generated  in  response  to  selected  inputs  and  execution 
conditions. 

Generic  Component.  A  generic  component  is  one  which  can  be  instantiated  in  a  number 
of  predefined  ways  so  that  each  occurrence  of  the  component  can  be  tailored  to  suit  a  par¬ 
ticular  usage.  For  example,  a  generic  component  which  provides  a  set  of  queue  handling 
routines  might  be  designed  so  that  it  can  be  instantiated  to  operate  on  queues  with  different 
message  formats. 

Global  Assertion.  Those  assertions  which  are  valid  for  the  whole  program  being  validated. 

Graph.  See  Directed  Graph. 

Incident.  During  testing,  any  event  that  occurs  during  the  execution  of  a  software  test  that 
requires  investigation. 

Incremental  Analysis.  Occurs  when  (partial)  analysis  may  be  performed  on  an  incomplete 
product  to  allow  early  feedback  on  the  development  of  that  product. 

Incremental  Development.  A  software  development  technique  in  which  requirements 
definition,  design,  implementation,  and  testing  occur  in  an  overlapping,  iterative  manner, 
resulting  in  an  incremental  completion  of  the  overall  software  product. 

Independent  Verification  and  Validation  (IV&V).  Verification  and  validation  of  a  soft¬ 
ware  product  by  an  organization  that  is  both  technically,  managerially,  and  financially  sep¬ 
arate  from  the  organization  responsible  for  developing  the  product  See  Validation  and 
Verification. 

Infeasible  Path.  A  sequence  of  program  statements  that  can  never  be  executed. 

Information  Flow  Analysis.  A  study  of  the  interdependencies  of  program  variables.  A 
given  variable  A  will  depend  on  another  variable  B  at  a  specific  point  in  the  program  if  the 
path  taken  in  reaching  that  point  is  such  that  the  value  of  A  depends  on  the  value  of  B. 

Inspection.  A  static  analysis  technique  that  relies  on  visual  examination  of  development 
products  to  detect  errors,  violations  of  development  standards,  and  other  problems.  Types 
include  code  inspections  and  design  inspections. 
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Instruction  Block.  A  sequence  of  statements  where  execution  of  the  first  statement  neces¬ 
sarily  leads  to  execution  of  the  last  statement 

Instrumentation.  Devices  or  instructions  installed  or  inserted  into  hardware  or  software  to 
monitor  the  operation  of  a  system  or  component. 

Integration.  The  process  of  combining  software  elements,  hardware  elements,  or  both  into 
an  overall  system. 

Integration  Testing.  Testing  in  which  software  components,  hardware  components,  or 
both  are  combined  and  tested  to  evaluate  the  interaction  between  them. 

Intervals.  Derived  from  a  directed  graph,  an  interval  is  defined  as  the  following:  An  inter¬ 
val  with  head  node  H  is  the  subgraph  containing  H  plus  any  nodes  that  can  be  reached  on 
a  path  from  H,  and  which  have  all  their  immediate  predecessors  in  the  interval.  First-order 
intervals  give  a  count  of  the  intervals  that  partition  the  graph  into  a  set  of  disjoint  compo¬ 
nents.  The  maximum  order  is  the  number  of  interval  iterations  required  to  reduce  a  graph 
to  a  single  node. 

Invocation.  The  transfer  of  control  to  an  entity  causing  it  to  be  activated. 

Linear  Code  Sequence  and  Jump  (LCSAJ)  Program  Units.  Sections  of  the  code 
through  which  the  flow  of  control  proceeds  sequentially  until  terminated  by  a  jump  in  the 
control  flow. 

Logarithmic  Poisson  Execution  Time  Model.  A  software  reliability  model  in  which  the 
failure  process  is  assumed  to  be  a  nonhomogeneous  Poisson  process  with  exponentially  de¬ 
creasing  failure  intensity. 

Maintainability.  (1)  The  probability  that  specified  unavailable  functions  can  be  repaired 
or  restored  to  their  operational  state  in  the  system’s  intended  maintenance  environment  dur¬ 
ing  a  specified  period  of  time.  (2)  The  average  effort  to  locate  and  fix  a  software  failure. 

Metric.  A  quantitative  measure  of  the  degree  to  which  a  system,  component,  or  process 
possesses  a  given  attribute. 

Module.  A  program  unit  that  is  discrete  and  identifiable  with  respect  to  compiling,  com¬ 
bining  with  other  units,  and  loading. 
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Node.  In  a  diagram,  a  point,  circle,  or  other  geometric  figure  used  to  represent  a  state,  event, 
or  other  item  of  interest. 

Operational  Testing.  Testing  performed  by  the  end  user  on  software  in  its  normal  operat¬ 
ing  environment. 

Parse.  To  determine  the  syntactic  structure  of  a  language  unit  by  decomposing  it  into  more 
elementary  subunits  and  establishing  the  relationships  among  the  subunits.  For  example,  to 
decompose  blocks  into  statements,  statements  into  expressions,  expressions  into  operators 
and  operands. 

Path.  In  software  engineering,  a  sequence  of  instructions  that  may  be  performed  in  the  ex¬ 
ecution  of  a  computer  program. 

Path  Analysis.  Analysis  of  a  computer  program  to  identify  all  possible  paths  through  the 
program,  to  detect  incomplete  paths,  or  to  discover  portions  of  the  program  that  are  not  on 
any  path. 

Path  Testing.  Testing  designed  to  execute  all  or  selected  paths  through  a  computer  pro¬ 
gram.  (Often  paths  through  the  program  are  grouped  into  a  finite  set  of  classes:  one  path 
from  each  class  is  tested.) 

Performance.  The  degree  to  which  a  system  or  component  accomplishes  its  designated 
functions  within  given  constraints,  such  as  speed,  accuracy,  or  memory  usage. 

Portability.  The  ease  with  which  a  system  or  component  can  be  transferred  from  one  hard¬ 
ware  or  software  environment  to  another. 

Pretty  Printing.  The  use  of  indentation,  blank  lines,  and  other  visual  clues  to  show  the  log¬ 
ical  structure  of  a  program. 

Program  Correctness.  (1)  The  extent  to  which  software  is  free  from  design  defects  and 
coding  defects;  that  is,  fault  free.  (2)  Extent  to  which  the  software  satisfies  its  specifications 
and  fulfills  the  user’s  mission  objects.  (3)  If  for  all  initial  states  that  belong  to  the  set  of  le¬ 
gitimate  initial  states,  the  program  P  terminates  with  a  final  state  that  belongs  to  the  set  of 
legitimate  final  states,  then  program  P  exhibits  program  correctness. 

Prototype.  A  limited  implementation  of  a  system  built  in  order  to  capture  or  validate  some 
aspects  of  a  system  design.  The  fundamental  concept  is  that  a  prototype  of  a  system  is  more 
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cheaply  or  more  quickly  constructed  than  the  actual  system.  Hence,  some  aspects  of  func¬ 
tion  or  performance  are  typically  sacrificed. 

Pseudo  Code.  A  combination  of  programming  language  constructs  and  natural  language 
used  to  express  a  computer  program  design. 

Quality.  (1)  The  degree  to  which  a  system,  component,  or  process  meets  specified  require¬ 
ments.  (2)  The  degree  to  which  a  system,  component,  or  process  meets  customer  or  user 
needs  or  expectations. 

Quality  Assurance.  (I)  A  planned  and  systematic  pattern  of  all  actions  necessary  to  pro¬ 
vide  adequate  confidence  that  the  item  or  product  conforms  to  established  technical  re¬ 
quirements.  (2)  A  set  of  activities  designed  to  evaluate  the  process  by  which  products  are 
developed  or  manufactured. 

Random  Testing.  An  essentially  black-box  testing  approach  in  which  a  program  is  tested 
by  randomly  choosing  a  subset  of  all  possible  input  values.  The  distribution  may  be  arbi¬ 
trary  or  may  attempt  to  accurately  reflect  the  distribution  of  inputs  in  the  application  envi¬ 
ronment. 

Regression  Testing:  Selective  retesting  to  detect  faults  introduced  during  modification  of 
a  system  or  system  component,  to  verify  that  modifications  have  not  caused  unintended  ad¬ 
verse  effects,  and  verify  that  a  modified  system  or  system  component  still  meets  its  speci¬ 
fied  requirements. 

Reliability.  The  ability  of  a  system  or  component  to  perform  its  required  functions  under 
stated  conditions  for  a  specified  period  of  time. 

Reliability  Model.  A  model  used  for  predicting,  estimating,  or  assessing  reliability. 

Reliability  Growth.  The  improvement  in  reliability  that  results  from  correction  of  faults. 

Requirement.  A  condition  or  capability  that  must  be  met  or  possessed  by  a  system  or  sys¬ 
tem  component  to  satisfy  a  contract,  standard,  specification,  or  other  formally  imposed  doc¬ 
ument.  The  set  of  all  requirements  forms  the  basis  for  subsequent  development  of  the 
system  or  system  component 
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Requirements  Specification.  A  document  that  specifies  the  requirements  for  a  system  or 
component.  Typically  included  are  functional  requirements,  performance  requirements,  in¬ 
terface  requirements,  design  requirements,  and  development  standards. 

Retesting.  See  Regression  Testing. 

Safety.  The  extent  to  which  the  program  is  protected  from  causing  a  specified  set  of  haz¬ 
ards. 

Scope.  The  range  within  which  an  identified  unit  displays  itself.  Scope  of  activity  refers  to 
the  boundaries  within  which  a  data  structure  or  program  element  remains  an  integral  unit. 
Scope  of  control  refers  to  the  submodules  in  a  program  that  potentially  may  execute  if  con¬ 
trol  is  given  to  a  cited  module.  Scope  of  error  denotes  the  set  of  submodules  that  are  poten¬ 
tially  affected  by  the  detection  of  a  fault  in  a  cited  module. 

Segment.  A  (logical)  segment,  or  decision-to-decision  path,  is  the  set  of  statements  in  a 
module  which  are  executed  as  a  result  of  the  evaluation  of  some  predicate  within  the  mod¬ 
ule.  It  begins  at  an  entry  or  decision  statement  and  ends  at  a  decision  statement  or  exit,  and 
should  be  thought  of  as  including  the  sensing  of  the  outcome  of  a  conditional  operation  and 
the  subsequent  statement  execution  up  to  and  including  the  computation  of  the  next  predi¬ 
cate  value,  but  not  including  its  evaluation. 

Self-Checking  Software.  Software  which  makes  an  explicit  attempt  to  determine  its  own 
correctness  and  to  proceed  accordingly. 

Simulation.  (1)  A  model  that  behaves  or  operates  like  a  system  when  provided  a  set  of  con¬ 
trolled  inputs.  (2)  The  process  of  developing  or  using  a  model  as  in  (1). 

Sizing.  The  process  of  estimating  the  amount  of  computer  storage  or  number  of  source  lines 
required  for  a  software  system  or  component 

Software.  Computer  programs,  procedures,  rules,  and  any  associated  documentation  per¬ 
taining  to  the  operation  of  a  computer  system. 

Software  Fault  Tree  Analysis.  A  form  of  fault  tree  analysis  used  for  analyzing  the  safety 
of  software  designs  or  code. 

Software  Quality.  ( 1 )  The  totality  of  features  and  characteristics  of  a  software  product  that 
bear  on  its  ability  to  satisfy  given  needs;  for  example,  conform  to  specifications.  (2)  The 
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degree  to  which  software  possesses  a  desired  combination  of  attributes.  (3)  The  composite 
characteristics  of  software  that  determine  the  degree  to  which  the  software  in  use  will  meet 
the  expectations  of  the  customer. 

Software  Reliability.  ( 1 )  The  probability  that  software  will  not  cause  the  failure  of  the  sys¬ 
tem  for  a  specified  time  under  specified  conditions.  The  probability  is  a  function  of  the  in¬ 
puts  to  and  use  of  the  system  as  well  as  a  function  of  the  existence  of  faults  in  the  software. 
The  inputs  to  the  system  determine  whether  existing  faults,  if  any,  are  encountered.  (2)  The 
ability  of  a  program  to  perform  a  required  function  under  stated  conditions  for  a  stated  pe¬ 
riod  of  time. 

Software  Reliability  Model.  A  model  used  for  predicting,  estimating,  or  assessing  soft¬ 
ware  reliability. 

Software  Science.  Software  Science  measures  the  complexity  of  a  software  module  by  cal¬ 
culations  based  on  the  incidence  of  references  to  operators  and  operands.  The  fundamental 
measures  calculated  are  program  vocabulary,  length,  and  volume. 

Specification.  A  document  that  prescribes  in  a  complete,  precise,  verifiable  manner,  the  re¬ 
quirements,  design,  behavior,  or  other  characteristics  of  a  system  or  system  component, 
and,  often,  the  procedures  for  determining  whether  these  provisions  have  been  satisfied. 

Specification  Language.  A  language,  often  a  machine-processable  combination  of  natural 
and  formal  language,  used  to  specify  the  requirements,  design,  behavior,  or  other  charac¬ 
teristics  of  a  system  or  system  component. 

Statement  Testing.  Testing  designed  to  execute  each  statement  in  a  computer  program. 

Static  Analyzer.  A  software  tool  that  aids  in  the  evaluation  of  a  computer  program  without 
executing  the  program.  Examples  include  syntax  checkers,  compilers,  cross-reference  gen¬ 
erators,  standards  enforcers,  and  flowcharters. 

Stress  Testing.  Testing  conducted  to  evaluate  a  system  or  component  at  or  beyond  the  lim¬ 
its  of  its  specified  requirements. 

Structural  Testing.  Testing  that  takes  into  account  the  internal  mechanism  of  a  system  or 
component  Types  include  branch  testing,  path  testing,  and  statement  testing. 
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Structured  Programming.  A  well-defined  software  development  technique  that  incorpo¬ 
rates  top-down  design  and  implementation  and  strict  use  of  structured  program  control  con¬ 
structs. 

Stub.  A  skeletal  or  special-purpose  implementation  of  a  software  module,  used  to  develop 
or  test  a  module  that  calls  or  is  otherwise  dependent  upon  it. 

Symbolic  Evaluation.  See  Symbolic  Execution. 

Symbolic  Execution.  A  software  analysis  technique  in  which  program  execution  is  simu¬ 
lated  using  symbols,  such  as  variable  names,  rather  than  actual  values  for  input  data,  and 
mathematical  expressions  involving  these  symbols. 

System  Testing.  Testing  conducted  on  a  complete,  integrated  system  to  evaluate  the  sys¬ 
tem’s  compliance  with  its  specified  requirements. 

Test.  A  unit  test  of  a  single  module  consists  of  (1 )  a  collection  of  settings  for  the  input  space 
of  the  module,  and  (2)  exactly  one  invocation  of  the  module.  A  unit  test  may  or  may  not 
include  the  effect  of  other  modules  which  are  invoked  by  the  module  undergoing  testing. 

Testbed.  An  environment  containing  the  hardware,  instrumentation,  simulators,  software 
tools,  and  other  support  elements  needed  to  conduct  a  test. 

Test  Case.  A  set  of  test  inputs,  execution  conditions,  and  expected  results  developed  for  a 
particular  objective,  such  as  to  exercise  a  particular  program  path  or  to  verify  compliance 
with  a  specific  requirements. 

Test  Data.  See  Test  Case. 

Test  Data  Generator.  A  software  tool  that  accepts  as  input  source  code,  test  criteria,  spec¬ 
ifications,  or  data  structure  definitions;  uses  these  inputs  to  generate  test  input  data;  and, 
sometimes,  determines  the  expected  results. 

Test  Driver.  A  program  that  directs  the  execution  of  another  program  against  a  collection 
of  test  data  sets.  Usually  the  test  driver  also  records  and  organizes  the  output  generated  as 
the  tests  are  run. 

Test  Management.  Management  procedures  designed  to  control  in  an  ordered  way  a  large 
and  evolving  amount  of  information  on  system  features  to  be  tested,  on  system  implemen¬ 
tation  plans,  and  on  test  results. 
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Test  Path.  The  specific  (sequence)  set  of  segments  that  is  traversed  as  the  result  of  a  unit 
test  operation  on  a  set  of  test  data.  A  module  can  have  many  test  paths. 

Test  Plan.  A  document  prescribing  the  approach  to  be  taken  for  intended  testing  activities. 
The  plan  typically  identifies  the  items  to  be  tested,  the  testing  to  be  performed,  test  sched¬ 
ules,  personnel  requirements,  reporting  requirements,  evaluation  criteria,  and  any  risks  re¬ 
quiring  contingency  planning. 

Test  Repeatability.  An  attribute  of  a  test  indicating  whether  the  same  results  are  produced 
each  time  the  test  is  conducted. 

Test  Log.  A  chronological  record  of  all  relevant  details  about  the  execution  of  a  test. 

Testability.  (1)  The  degree  to  which  a  system  or  component  facilitates  the  establishment 
of  test  criteria  and  the  performance  of  tests  to  determine  whether  these  criteria  have  been 
met  (2)  The  degree  to  which  a  requirement  is  stated  in  terms  that  permit  establishment  of 
test  criteria  and  performance  of  tests  to  determine  whether  those  criteria  have  been  met. 

Testing.  The  process  of  exercising  or  evaluating  a  system  or  system  component  by  manual 
or  automated  means  to  verify  that  it  satisfies  specified  requirements  or  to  identify  differ¬ 
ences  between  expected  and  actual  results. 

Timing  Analyzer.  A  software  tool  that  estimates  or  measures  the  execution  time  of  a  com¬ 
puter  program  or  portion  of  a  computer  program,  either  by  summing  up  the  execution  times 
of  the  instructions  along  the  specified  paths  or  by  inserting  probes  at  specified  points  in  the 
program  and  measuring  the  execution  time  between  probes. 

Top-Down  Testing.  A  systematic  testing  philosophy  which  seeks  to  first  test  those  mod¬ 
ules  at  the  top  of  the  invocation  structure. 

Trace.  A  record  of  the  execution  of  a  computer  program,  showing  the  sequences  of  instruc¬ 
tions  executed,  the  names  and  values  of  variables,  or  both. 

Traceability.  The  degree  to  which  a  relationship  can  be  established  between  two  or  more 
products  of  the  development  process,  especially  products  having  a  predecessor-successor 
or  master-subordinate  relationship  to  one  another;  for  example,  the  degree  to  which  the  re¬ 
quirements  and  design  of  a  given  software  component  match. 
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Unit.  A  separately  testable  element  specified  in  the  design  of  a  computer  software  compo¬ 
nent. 

Unit  Testing.  Testing  of  individual  hardware  or  software  units  or  groups  of  related  units. 
See  also  Integration  Testing  and  System  Testing. 

Unreachabiiity.  A  statement  (or  segment)  is  unreachable  if  there  is  no  logically  obtainable 
set  of  input-space  settings  which  can  cause  the  statement  (or  segment)  to  be  traversed. 

Validation.  The  process  of  evaluating  a  system  or  component  during  or  at  the  end  of  the 
development  process  to  determine  whether  it  satisfies  specified  requirements. 

Verification.  (1)  The  process  of  evaluating  a  system  or  component  to  determine  whether 
the  products  of  a  given  development  phase  satisfy  the  conditions  imposed  at  the  start  of  that 
phase.  (2)  Formal  proof  of  correctness. 
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